{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Multi-layer Graph Convolutional Network (GCN) with first-order filters,来源:http://tkipf.github.io/graph-convolutional-networks/](images/gcn_web.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. GCN是什么\n",
    "\n",
    "图卷积神经网络(Graph Convolution Networks, GCN)跟CNN一样是特征提取的工具，CNN在处理规则数据结构(如图片等)方面非常强大。\n",
    "\n",
    "![图像矩阵示意图（Euclidean Structure）](images/cnn_sample.jpg)\n",
    "\n",
    "但是在现实世界中，很多数据结构是不规则的，典型的就是图结构，如社交网络、知识图谱等，GNN就比较擅长处理这类数据。\n",
    "\n",
    "![社交网络拓扑示意（Non Euclidean Structure）](images/social_non_struct.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "本文主要通过一个完整的GCN对论文进行分类的例子，来展示GCN的工作过程和原理。\n",
    "\n",
    "这里我们使用的Cora数据集，该数据集由2708篇论文的特征、分类以及它们之间引用关系的5429条边组成，这些论文的类型被划分为7个类别:Case\\_Based、Genetic\\_Algorithms、Neural\\_Networks、Probabilistic\\_Methods、Reinforcement\\_Learning、Rule\\_Learning、Theory。\n",
    "\n",
    "最终实现的目标是，输入一篇论文的特征，就可以输出该论文属于哪个分类。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. 数据集-Cora Dataset\n",
    "\n",
    "## 2.1 下载地址\n",
    "\n",
    "https://linqs-data.soe.ucsc.edu/public/lbc/cora.tgz\n",
    "\n",
    "## 2.2 数据内容\n",
    "\n",
    "Cora Dataset是对Machine Learning Paper进行分类的数据集，它包含三个文件:\n",
    "\n",
    "-- README: 对数据集的介绍;\n",
    "\n",
    "-- cora.cites: 论文之间的引用关系图。文件中每行包含两个Paper ID， 第一个ID是被引用的Paper ID； 第二个是引用的Paper ID。格式如下: <ID of cited paper> <ID of citing paper>\n",
    "\n",
    "-- cora.content: 包含了2708篇Paper的信息，每行的数据格式如下: <paper_id> <word_attributes>+ <class_label>。paper id是论文的唯一标识；word_attributes是是一个维度为1433的词向量，词向量的每个元素对应一个词，0表示该元素对应的词不在Paper中，1表示该元素对应的词在Paper中。class_label是论文的类别，每篇Paper被映射到如下7个分类之一: Case_Based、Genetic_Algorithms、Neural_Networks、Probabilistic_Methods、Reinforcement_Learning、Rule_Learning、Theory。\n",
    "    \n",
    "先看看Cora Dataset中的数据是什么样的..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cora Contents’s Row: 2708, Col: 1435\n",
      "=============================================\n",
      "features:   1     2     3     4     5     6     7     8     9     10    ...  1424  \\\n",
      "0     0     0     0     0     0     0     0     0     0     0  ...     0   \n",
      "1     0     0     0     0     0     0     0     0     0     0  ...     0   \n",
      "2     0     0     0     0     0     0     0     0     0     0  ...     0   \n",
      "\n",
      "   1425  1426  1427  1428  1429  1430  1431  1432  1433  \n",
      "0     0     0     1     0     0     0     0     0     0  \n",
      "1     0     1     0     0     0     0     0     0     0  \n",
      "2     0     0     0     0     0     0     0     0     0  \n",
      "\n",
      "[3 rows x 1433 columns]\n",
      "=============================================\n",
      "labels:0           Neural_Networks\n",
      "1             Rule_Learning\n",
      "2    Reinforcement_Learning\n",
      "Name: 1434, dtype: object\n",
      "=============================================\n",
      "labels one hot:   Neural_Networks  Reinforcement_Learning  Rule_Learning\n",
      "0                1                       0              0\n",
      "1                0                       0              1\n",
      "2                0                       1              0\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# 导入数据：分隔符为Tab\n",
    "raw_data_content = pd.read_csv('data/cora/cora.content',sep = '\\t',header = None)\n",
    "\n",
    "# [2708 * 1435]\n",
    "(row, col) = raw_data_content.shape\n",
    "print(\"Cora Contents’s Row: {}, Col: {}\".format(row, col))\n",
    "\n",
    "print(\"=============================================\")\n",
    "\n",
    "# 每行是1435维的向量，第一维是论文的ID，最后一维是论文的Label\n",
    "raw_data_sample = raw_data_content.head(3)\n",
    "features_sample =raw_data_sample.iloc[:,1:-1]\n",
    "labels_sample = raw_data_sample.iloc[:, -1]\n",
    "labels_onehot_sample = pd.get_dummies(labels_sample)\n",
    "\n",
    "\n",
    "print(\"features:{}\".format(features_sample))\n",
    "\n",
    "print(\"=============================================\")\n",
    "\n",
    "print(\"labels:{}\".format(labels_sample))\n",
    "\n",
    "\n",
    "print(\"=============================================\")\n",
    "\n",
    "print(\"labels one hot:{}\".format(labels_onehot_sample))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cora Cites’s Row: 5429, Col: 2\n",
      "           0        1\n",
      "0         35     1033\n",
      "1         35   103482\n",
      "2         35   103515\n",
      "3         35  1050679\n",
      "4         35  1103960\n",
      "...      ...      ...\n",
      "5424  853116    19621\n",
      "5425  853116   853155\n",
      "5426  853118  1140289\n",
      "5427  853155   853118\n",
      "5428  954315  1155073\n",
      "\n",
      "[5429 rows x 2 columns]\n",
      "=============================================\n",
      "  (163, 402)\t1.0\n",
      "  (163, 659)\t1.0\n",
      "  (163, 1696)\t1.0\n",
      "  (163, 2295)\t1.0\n",
      "  (163, 1274)\t1.0\n",
      "  (163, 1286)\t1.0\n",
      "  (163, 1544)\t1.0\n",
      "  (163, 2600)\t1.0\n",
      "  (163, 2363)\t1.0\n",
      "  (163, 1905)\t1.0\n",
      "  (163, 1611)\t1.0\n",
      "  (163, 141)\t1.0\n",
      "  (163, 1807)\t1.0\n",
      "  (163, 1110)\t1.0\n",
      "  (163, 174)\t1.0\n",
      "  (163, 2521)\t1.0\n",
      "  (163, 1792)\t1.0\n",
      "  (163, 1675)\t1.0\n",
      "  (163, 1334)\t1.0\n",
      "  (163, 813)\t1.0\n",
      "  (163, 1799)\t1.0\n",
      "  (163, 1943)\t1.0\n",
      "  (163, 2077)\t1.0\n",
      "  (163, 765)\t1.0\n",
      "  (163, 769)\t1.0\n",
      "  :\t:\n",
      "  (2228, 1093)\t1.0\n",
      "  (2228, 1094)\t1.0\n",
      "  (2228, 2068)\t1.0\n",
      "  (2228, 2085)\t1.0\n",
      "  (2694, 2331)\t1.0\n",
      "  (617, 226)\t1.0\n",
      "  (422, 1691)\t1.0\n",
      "  (2142, 2096)\t1.0\n",
      "  (1477, 1252)\t1.0\n",
      "  (1485, 1252)\t1.0\n",
      "  (2185, 2109)\t1.0\n",
      "  (2117, 2639)\t1.0\n",
      "  (1211, 1247)\t1.0\n",
      "  (1884, 745)\t1.0\n",
      "  (1884, 1886)\t1.0\n",
      "  (1884, 1902)\t1.0\n",
      "  (1885, 745)\t1.0\n",
      "  (1885, 1884)\t1.0\n",
      "  (1885, 1886)\t1.0\n",
      "  (1885, 1902)\t1.0\n",
      "  (1886, 745)\t1.0\n",
      "  (1886, 1902)\t1.0\n",
      "  (1887, 2258)\t1.0\n",
      "  (1902, 1887)\t1.0\n",
      "  (837, 1686)\t1.0\n"
     ]
    }
   ],
   "source": [
    "raw_data_cites = pd.read_csv('data/cora/cora.cites',sep = '\\t',header = None)\n",
    "\n",
    "# [5429 * 2]\n",
    "(row, col) = raw_data_cites.shape\n",
    "\n",
    "print(\"Cora Cites’s Row: {}, Col: {}\".format(row, col))\n",
    "\n",
    "print(\"=============================================\")\n",
    "\n",
    "raw_data_cites_sample = raw_data_cites.head(10)\n",
    "\n",
    "print(raw_data_cites_sample)\n",
    "\n",
    "print(\"=============================================\")\n",
    "\n",
    "# raw_data_cites.head(10).values.flatten().tolist()\n",
    "\n",
    "\n",
    "# Convert Cite to adj matrix\n",
    "idx = np.array(raw_data_content.iloc[:, 0], dtype=np.int32)\n",
    "idx_map = {j: i for i, j in enumerate(idx)}\n",
    "\n",
    "\n",
    "edge_indexs = np.array(list(map(idx_map.get, raw_data_cites.values.flatten())), dtype=np.int32)\n",
    "edge_indexs = edge_indexs.reshape(raw_data_cites.shape)\n",
    "adjacency = sp.coo_matrix((np.ones(len(edge_indexs)),\n",
    "            (edge_indexs[:, 0], edge_indexs[:, 1])),\n",
    "            shape=(edge_indexs.shape[0], edge_indexs.shape[0]), dtype=\"float32\")\n",
    "\n",
    "print(adjacency)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. 构造训练集、测试集和验证集\n",
    "\n",
    "这里使用[0, 150)个数据作为训练集合，[150, 500)个数据作为验证集，[500, 2708)个数据作为测试集，实现上使用掩码(train_mask、val_mask、test_mask)的形式来区分训练集、验证集和测试集。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_index = np.arange(150)\n",
    "val_index = np.arange(150, 500)\n",
    "test_index = np.arange(500, 2708)\n",
    "\n",
    "train_mask = np.zeros(edge_indexs.shape[0], dtype = np.bool)\n",
    "val_mask = np.zeros(edge_indexs.shape[0], dtype = np.bool)\n",
    "test_mask = np.zeros(edge_indexs.shape[0], dtype = np.bool)\n",
    "\n",
    "train_mask[train_index] = True\n",
    "val_mask[val_index] = True\n",
    "test_mask[test_index] = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. GCN核心网络模型\n",
    "\n",
    "图(Graph)其实数据结构中最重要的概念之一，对，没错，图神经网的图(Graph)跟数据结构中的图(Graph)是一回事。假设神经网络的输入图(Graph)包含N个节点(Node)，每个节点有d个特征，则所有这些节点的特征组成一个Nxd维的矩阵X；两个节点间的邻接关系组成一个NxN的邻接矩阵A(adjacency),则X和A就构成了图神经网络的输入。\n",
    "\n",
    "## 4.1 核心公式：\n",
    "\n",
    "GCN神经网络相邻两层之间传播的核心公式如下：\n",
    "\n",
    "$$H^{l+1}=\\sigma\\left(\\tilde{D}^{-\\frac{1}{2}} \\tilde{A} \\tilde{D}^{-\\frac{1}{2}} H^{(l)} W^{(l)}\\right) \\tag{1}$$\n",
    "\n",
    "其中：\n",
    "\n",
    "$\\tilde{A} = A + I$是图G的邻接矩阵加上自连接，I是单位矩阵；\n",
    "\n",
    "$\\tilde{D}$是度矩阵(Degree Matrix), 计算公式为：$\\tilde{D}_{i i}=\\sum_{j} \\tilde{A}_{i j}$；\n",
    "\n",
    "H是每一层的特征；\n",
    "\n",
    "W是神经网络的待训练的权重参数；\n",
    "\n",
    "$\\sigma$是激活函数，常用的激活函数有softmax、relu等；\n",
    "\n",
    "我们先看下GCN网络的传播过程，再反过来看公式中详细含义。\n",
    "\n",
    "## 4.2 网络传播过程\n",
    "\n",
    "如下所示的5个Node组成的无向图的图结构。\n",
    "\n",
    "![](images/graph.png)\n",
    "\n",
    "它对应的邻接矩阵A为：\n",
    "\n",
    "$$\n",
    "A = \\left[ \\begin{matrix} 0 & 1 & 0 & 0 & 1\\\\ 1 & 0 & 1 & 1 & 0 \\\\ 0 & 1 & 0 & 1 & 0 \\\\ 0 & 1 & 1 & 0 & 1 \\\\ 1 & 0 & 0 & 1 & 0 \\end{matrix} \\right]\n",
    "$$\n",
    "\n",
    "在邻接矩阵基础上加上自环，即与单位矩阵I相加，得到：\n",
    "\n",
    "$$\n",
    "\\tilde{A} = \\left[ \\begin{matrix} 1 & 1 & 0 & 0 & 1\\\\ 1 & 1 & 1 & 1 & 0 \\\\ 0 & 1 & 1 & 1 & 0 \\\\ 0 & 1 & 1 & 1 & 1 \\\\ 1 & 0 & 0 & 1 & 1 \\end{matrix} \\right]\n",
    "$$\n",
    "\n",
    "简单期间，这里假设每个Graph中每个Node的特征都是一维的，$X=[1, 2, 3, 4, 5]^T$,\n",
    "\n",
    "$$ \n",
    "\\tilde{A} * X = \\left[ \\begin{matrix} 1 * 1 + 1 * 2 + 0 * 3 + 0 * 4 + 1 * 5 \\\\ 1 * 1 + 1 * 2 + 1 * 3 + 1 * 4 + 0 * 5 \\\\ 0 * 1 + 1 * 2 + 1 * 3 + 1 * 4 + 0 * 5 \\\\ 0 * 1 + 1 * 2 + 1 * 3 + 1 * 4 + 1 * 5 \\\\ 1 * 1 + 0 * 2 + 0 * 3 + 1 * 4 + 1 * 5 \\\\ \\end{matrix} \\right]\n",
    "$$\n",
    "\n",
    "从上式看到了什么，对，就是**各个节点都将与其一阶相邻节点的信息融合到自己的节点中，这也是公式(1)的本质所在。神经网络传播的过程，实际上就是图(Graph)中各个节点(Node)不断聚合邻居节点信息的过程。** 通过两次连乘$\\tilde{A} * \\tilde{A} * X$也就实现各个Node融合自己二阶邻居节点的信息。\n",
    "\n",
    "到这里我们应该就可以理解**为什么要将邻接矩阵加上单位矩阵**。\n",
    "\n",
    "> 因为邻接矩阵的对角线的值都是0，所以如果用邻接矩阵直接与特征矩阵相乘，就将节点自身的信息丢失了。所以为了保留节点(Node)的自身特征，需要将邻接矩阵加上单位矩阵。\n",
    "\n",
    "既然$\\tilde{A} * X$就可以实现图网络中的信息聚合，**那么$\\tilde{D}^{-\\frac{1}{2}} \\tilde{A} \\tilde{D}^{-\\frac{1}{2}}$ 是什么意思？**\n",
    "\n",
    "这里只从直觉上说明这个问题，(事实上，我还没有来得及去详细推导公式，后续有时间补上，嘿嘿)\n",
    "\n",
    "> 假设节点A的邻居只有B，为了对A进行信息聚合，最直接的方法就是平均贡献，即： New_A = 0.5* A + 0.5 * B，这样的做法看起来很合理，但是如果B的邻居非常多(极端情况是，它与图Graph上的所有其它节点都有连接)，经过特征聚合之后，图(Graph)上许多的特征就会非常相似，因此需要考虑节点(Node)的度，不要让度过大的节点(Node)贡献过大。\n",
    "\n",
    "\n",
    "两层的GCN网络实现如下:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "\n",
    "import tensorflow as tf\n",
    "\n",
    "class GraphConvolution(tf.keras.layers.Layer):\n",
    "    \"\"\"Basic graph convolution layer as in https://arxiv.org/abs/1609.02907\"\"\"\n",
    "    def __init__(self, units, support=1,\n",
    "                 activation=None,\n",
    "                 use_bias=True\n",
    "                 kernel_initializer='glorot_uniform',\n",
    "                 bias_initializer='zeros',\n",
    "                 kernel_regularizer=None,\n",
    "                 bias_regularizer=None,\n",
    "                ):\n",
    "        \n",
    "        super(GraphConvolution, self).__init__()\n",
    "        self.units = units\n",
    "        self.use_bias = use_bias\n",
    "        self.kernel_initializer = kernel_initializer\n",
    "        self.bias_initializer = bias_initializer\n",
    "        self.kernel_regularizer = kernel_regularizer\n",
    "        self.bias_regularizer = bias_regularizer\n",
    "        \n",
    "        self.supports_masking = True\n",
    "\n",
    "        self.support = support\n",
    "        assert support >= 1\n",
    "\n",
    "    def build(self, input_shapes):\n",
    "        features_shape = input_shapes[0]\n",
    "        assert len(features_shape) == 2\n",
    "        input_dim = features_shape[1]\n",
    "\n",
    "        self.kernel = self.add_weight(shape = (input_dim * self.support, self.units),\n",
    "                                      initializer = self.kernel_initializer,\n",
    "                                      name = 'kernel',\n",
    "                                      regularizer = self.kernel_regularizer)\n",
    "        if self.use_bias:\n",
    "            self.bias = self.add_weight(shape=(self.units,),\n",
    "                                        initializer=self.bias_initializer,\n",
    "                                        name='bias',\n",
    "                                       regularizer = self.kernel_regularizer)\n",
    "        else:\n",
    "            self.bias = None\n",
    "            \n",
    "        self.built = True\n",
    "\n",
    "    def call(self, inputs, mask=None):\n",
    "        features = inputs[0]\n",
    "        basis = inputs[1:]\n",
    "\n",
    "        supports = list()\n",
    "        for i in range(self.support):\n",
    "            supports.append(K.dot(basis[i], features))\n",
    "        supports = K.concatenate(supports, axis=1)\n",
    "        output = K.dot(supports, self.kernel)\n",
    "\n",
    "        if self.bias:\n",
    "            output += self.bias\n",
    "            \n",
    "        return self.activation(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. 网络训练过程\n",
    "\n",
    "## 5.1 准备训练数据\n",
    "\n",
    "数据处理的细节前面都大概提过，这里需要注意的是，在数据处理的过程中，还需要对每个Node的Feature做归一化处理。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Process data ...\n",
      "Loading cora dataset...\n",
      "Dataset has 2708 nodes, 2708 edges, 1433 features.\n"
     ]
    }
   ],
   "source": [
    "from graph import GraphConvolutionLayer, GraphConvolutionModel\n",
    "from dataset import CoraData\n",
    "\n",
    "import time\n",
    "import tensorflow as tf\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "dataset = CoraData()\n",
    "features, labels, adj, train_mask, val_mask, test_mask = dataset.data()\n",
    "\n",
    "graph = [features, adj]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.2 Loss计算\n",
    "\n",
    "Loss函数中，只对训练数据(train_mask为True)进行Loss计算。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss_object = tf.keras.losses.CategoricalCrossentropy(from_logits=True)\n",
    "\n",
    "def loss(model, x, y, train_mask, training):\n",
    "\n",
    "    y_ = model(x, training=training)\n",
    "\n",
    "    test_mask_logits = tf.gather_nd(y_, tf.where(train_mask))\n",
    "    masked_labels = tf.gather_nd(y, tf.where(train_mask))\n",
    "\n",
    "    return loss_object(y_true=masked_labels, y_pred=test_mask_logits)\n",
    "\n",
    "\n",
    "def grad(model, inputs, targets, train_mask):\n",
    "    with tf.GradientTape() as tape:\n",
    "        loss_value = loss(model, inputs, targets, train_mask, training=True)\n",
    "    \n",
    "    return loss_value, tape.gradient(loss_value, model.trainable_variables)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.3 实际训练过程 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layer graph_convolution_model_5 is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2.  The layer has dtype float32 because it's dtype defaults to floatx.\n",
      "\n",
      "If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n",
      "\n",
      "To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n",
      "\n",
      "Epoch 0 loss=1.9472886323928833 accuracy=0.4066666666666667 val_acc=0.3142857142857143 test_acc=0.2817028985507246\n",
      "Epoch 1 loss=1.9314587116241455 accuracy=0.4866666666666667 val_acc=0.3742857142857143 test_acc=0.33106884057971014\n",
      "Epoch 2 loss=1.9133251905441284 accuracy=0.5 val_acc=0.38571428571428573 test_acc=0.34782608695652173\n",
      "Epoch 3 loss=1.8908278942108154 accuracy=0.5266666666666666 val_acc=0.3942857142857143 test_acc=0.3496376811594203\n",
      "Epoch 4 loss=1.8662141561508179 accuracy=0.5533333333333333 val_acc=0.3942857142857143 test_acc=0.3423913043478261\n",
      "Epoch 5 loss=1.8400791883468628 accuracy=0.56 val_acc=0.38285714285714284 test_acc=0.3401268115942029\n",
      "Epoch 6 loss=1.8119205236434937 accuracy=0.5866666666666667 val_acc=0.38571428571428573 test_acc=0.33605072463768115\n",
      "Epoch 7 loss=1.78205144405365 accuracy=0.6066666666666667 val_acc=0.37714285714285717 test_acc=0.33016304347826086\n",
      "Epoch 8 loss=1.751450777053833 accuracy=0.6066666666666667 val_acc=0.38857142857142857 test_acc=0.33016304347826086\n",
      "Epoch 9 loss=1.7200360298156738 accuracy=0.6066666666666667 val_acc=0.4057142857142857 test_acc=0.3342391304347826\n",
      "Epoch 10 loss=1.6870578527450562 accuracy=0.6333333333333333 val_acc=0.42 test_acc=0.3428442028985507\n",
      "Epoch 11 loss=1.6523456573486328 accuracy=0.64 val_acc=0.43142857142857144 test_acc=0.34646739130434784\n",
      "Epoch 12 loss=1.616371512413025 accuracy=0.6333333333333333 val_acc=0.44 test_acc=0.35190217391304346\n",
      "Epoch 13 loss=1.579743504524231 accuracy=0.64 val_acc=0.44857142857142857 test_acc=0.360054347826087\n",
      "Epoch 14 loss=1.5426799058914185 accuracy=0.64 val_acc=0.4542857142857143 test_acc=0.36594202898550726\n",
      "Epoch 15 loss=1.5049867630004883 accuracy=0.6466666666666666 val_acc=0.46285714285714286 test_acc=0.3686594202898551\n",
      "Epoch 16 loss=1.466316819190979 accuracy=0.6666666666666666 val_acc=0.46285714285714286 test_acc=0.37273550724637683\n",
      "Epoch 17 loss=1.4266818761825562 accuracy=0.6733333333333333 val_acc=0.4857142857142857 test_acc=0.37726449275362317\n",
      "Epoch 18 loss=1.3862168788909912 accuracy=0.6866666666666666 val_acc=0.5 test_acc=0.3808876811594203\n",
      "Epoch 19 loss=1.3451327085494995 accuracy=0.7266666666666667 val_acc=0.5114285714285715 test_acc=0.39221014492753625\n",
      "Epoch 20 loss=1.3035770654678345 accuracy=0.7533333333333333 val_acc=0.5257142857142857 test_acc=0.396286231884058\n",
      "Epoch 21 loss=1.2615602016448975 accuracy=0.7866666666666666 val_acc=0.5342857142857143 test_acc=0.40806159420289856\n",
      "Epoch 22 loss=1.2191429138183594 accuracy=0.8 val_acc=0.5457142857142857 test_acc=0.40851449275362317\n",
      "Epoch 23 loss=1.1763759851455688 accuracy=0.82 val_acc=0.56 test_acc=0.41893115942028986\n",
      "Epoch 24 loss=1.133314609527588 accuracy=0.82 val_acc=0.5742857142857143 test_acc=0.4316123188405797\n",
      "Epoch 25 loss=1.09010648727417 accuracy=0.8666666666666667 val_acc=0.5771428571428572 test_acc=0.4429347826086957\n",
      "Epoch 26 loss=1.0468487739562988 accuracy=0.8666666666666667 val_acc=0.5942857142857143 test_acc=0.452445652173913\n",
      "Epoch 27 loss=1.0036686658859253 accuracy=0.9 val_acc=0.6028571428571429 test_acc=0.46195652173913043\n",
      "Epoch 28 loss=0.9607122540473938 accuracy=0.9466666666666667 val_acc=0.6171428571428571 test_acc=0.47101449275362317\n",
      "Epoch 29 loss=0.9181469678878784 accuracy=0.9666666666666667 val_acc=0.6257142857142857 test_acc=0.47690217391304346\n",
      "Epoch 30 loss=0.8761056661605835 accuracy=0.9666666666666667 val_acc=0.6371428571428571 test_acc=0.48777173913043476\n",
      "Epoch 31 loss=0.8347143530845642 accuracy=0.9666666666666667 val_acc=0.6485714285714286 test_acc=0.4986413043478261\n",
      "Epoch 32 loss=0.79410320520401 accuracy=0.9666666666666667 val_acc=0.6514285714285715 test_acc=0.5140398550724637\n",
      "Epoch 33 loss=0.7544015645980835 accuracy=0.9666666666666667 val_acc=0.6514285714285715 test_acc=0.5253623188405797\n",
      "Epoch 34 loss=0.7157045602798462 accuracy=0.9666666666666667 val_acc=0.6628571428571428 test_acc=0.5335144927536232\n",
      "Epoch 35 loss=0.6780853271484375 accuracy=0.9666666666666667 val_acc=0.6657142857142857 test_acc=0.5443840579710145\n",
      "Epoch 36 loss=0.6416434049606323 accuracy=0.98 val_acc=0.6771428571428572 test_acc=0.5498188405797102\n",
      "Epoch 37 loss=0.6064579486846924 accuracy=0.9866666666666667 val_acc=0.6685714285714286 test_acc=0.5588768115942029\n",
      "Epoch 38 loss=0.5725471377372742 accuracy=1.0 val_acc=0.6771428571428572 test_acc=0.5706521739130435\n",
      "Epoch 39 loss=0.5399670600891113 accuracy=1.0 val_acc=0.6771428571428572 test_acc=0.5765398550724637\n",
      "Epoch 40 loss=0.5087469220161438 accuracy=1.0 val_acc=0.68 test_acc=0.5806159420289855\n",
      "Epoch 41 loss=0.4788845181465149 accuracy=1.0 val_acc=0.68 test_acc=0.5869565217391305\n",
      "Epoch 42 loss=0.4503992795944214 accuracy=1.0 val_acc=0.6828571428571428 test_acc=0.5892210144927537\n",
      "Epoch 43 loss=0.4233132302761078 accuracy=1.0 val_acc=0.6857142857142857 test_acc=0.5932971014492754\n",
      "Epoch 44 loss=0.3976200222969055 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.5960144927536232\n",
      "Epoch 45 loss=0.3733169138431549 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.5973731884057971\n",
      "Epoch 46 loss=0.3503628969192505 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.6000905797101449\n",
      "Epoch 47 loss=0.32872718572616577 accuracy=1.0 val_acc=0.6942857142857143 test_acc=0.6014492753623188\n",
      "Epoch 48 loss=0.3083726167678833 accuracy=1.0 val_acc=0.7 test_acc=0.6023550724637681\n",
      "Epoch 49 loss=0.28924670815467834 accuracy=1.0 val_acc=0.7028571428571428 test_acc=0.6032608695652174\n",
      "Epoch 50 loss=0.27130240201950073 accuracy=1.0 val_acc=0.7057142857142857 test_acc=0.6023550724637681\n",
      "Epoch 51 loss=0.2544911503791809 accuracy=1.0 val_acc=0.7028571428571428 test_acc=0.6032608695652174\n",
      "Epoch 52 loss=0.23875992000102997 accuracy=1.0 val_acc=0.7057142857142857 test_acc=0.6041666666666666\n",
      "Epoch 53 loss=0.22406704723834991 accuracy=1.0 val_acc=0.7085714285714285 test_acc=0.6059782608695652\n",
      "Epoch 54 loss=0.21036122739315033 accuracy=1.0 val_acc=0.7085714285714285 test_acc=0.6077898550724637\n",
      "Epoch 55 loss=0.1975751519203186 accuracy=1.0 val_acc=0.7057142857142857 test_acc=0.6073369565217391\n",
      "Epoch 56 loss=0.18565499782562256 accuracy=1.0 val_acc=0.7028571428571428 test_acc=0.6064311594202898\n",
      "Epoch 57 loss=0.17455774545669556 accuracy=1.0 val_acc=0.7028571428571428 test_acc=0.6073369565217391\n",
      "Epoch 58 loss=0.16423100233078003 accuracy=1.0 val_acc=0.7028571428571428 test_acc=0.6082427536231884\n",
      "Epoch 59 loss=0.15462662279605865 accuracy=1.0 val_acc=0.7 test_acc=0.6096014492753623\n",
      "Epoch 60 loss=0.14569756388664246 accuracy=1.0 val_acc=0.7 test_acc=0.6109601449275363\n",
      "Epoch 61 loss=0.13739608228206635 accuracy=1.0 val_acc=0.7 test_acc=0.6109601449275363\n",
      "Epoch 62 loss=0.12968382239341736 accuracy=1.0 val_acc=0.6942857142857143 test_acc=0.6127717391304348\n",
      "Epoch 63 loss=0.12251587212085724 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.6127717391304348\n",
      "Epoch 64 loss=0.11585451662540436 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6118659420289855\n",
      "Epoch 65 loss=0.10966223478317261 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6114130434782609\n",
      "Epoch 66 loss=0.10390333086252213 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6114130434782609\n",
      "Epoch 67 loss=0.09854617714881897 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6109601449275363\n",
      "Epoch 68 loss=0.09356522560119629 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6105072463768116\n",
      "Epoch 69 loss=0.08893042057752609 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6100543478260869\n",
      "Epoch 70 loss=0.08461488038301468 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.6100543478260869\n",
      "Epoch 71 loss=0.08059524744749069 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.6105072463768116\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 72 loss=0.07684874534606934 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.6100543478260869\n",
      "Epoch 73 loss=0.0733552798628807 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.6096014492753623\n",
      "Epoch 74 loss=0.07009640336036682 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.6100543478260869\n",
      "Epoch 75 loss=0.06705603748559952 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.6096014492753623\n",
      "Epoch 76 loss=0.06421645730733871 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.6091485507246377\n",
      "Epoch 77 loss=0.06155867129564285 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.6086956521739131\n",
      "Epoch 78 loss=0.05906983092427254 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.6086956521739131\n",
      "Epoch 79 loss=0.05673719570040703 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.6086956521739131\n",
      "Epoch 80 loss=0.054548606276512146 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6091485507246377\n",
      "Epoch 81 loss=0.052494876086711884 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6091485507246377\n",
      "Epoch 82 loss=0.050564948469400406 accuracy=1.0 val_acc=0.6857142857142857 test_acc=0.6082427536231884\n",
      "Epoch 83 loss=0.04874930530786514 accuracy=1.0 val_acc=0.6857142857142857 test_acc=0.6082427536231884\n",
      "Epoch 84 loss=0.04704001545906067 accuracy=1.0 val_acc=0.6857142857142857 test_acc=0.6086956521739131\n",
      "Epoch 85 loss=0.04542906954884529 accuracy=1.0 val_acc=0.6857142857142857 test_acc=0.6082427536231884\n",
      "Epoch 86 loss=0.04390912503004074 accuracy=1.0 val_acc=0.6857142857142857 test_acc=0.6077898550724637\n",
      "Epoch 87 loss=0.04247550293803215 accuracy=1.0 val_acc=0.6828571428571428 test_acc=0.6082427536231884\n",
      "Epoch 88 loss=0.04112052172422409 accuracy=1.0 val_acc=0.6828571428571428 test_acc=0.6073369565217391\n",
      "Epoch 89 loss=0.03983796760439873 accuracy=1.0 val_acc=0.6828571428571428 test_acc=0.6073369565217391\n",
      "Epoch 90 loss=0.03862294182181358 accuracy=1.0 val_acc=0.6828571428571428 test_acc=0.6073369565217391\n",
      "Epoch 91 loss=0.03747102618217468 accuracy=1.0 val_acc=0.6857142857142857 test_acc=0.6064311594202898\n",
      "Epoch 92 loss=0.03637789562344551 accuracy=1.0 val_acc=0.6857142857142857 test_acc=0.6059782608695652\n",
      "Epoch 93 loss=0.035339970141649246 accuracy=1.0 val_acc=0.6857142857142857 test_acc=0.605072463768116\n",
      "Epoch 94 loss=0.03435278683900833 accuracy=1.0 val_acc=0.6857142857142857 test_acc=0.605072463768116\n",
      "Epoch 95 loss=0.03341297432780266 accuracy=1.0 val_acc=0.6857142857142857 test_acc=0.6041666666666666\n",
      "Epoch 96 loss=0.03251757472753525 accuracy=1.0 val_acc=0.6857142857142857 test_acc=0.603713768115942\n",
      "Epoch 97 loss=0.03166373074054718 accuracy=1.0 val_acc=0.6857142857142857 test_acc=0.6032608695652174\n",
      "Epoch 98 loss=0.03084862045943737 accuracy=1.0 val_acc=0.6857142857142857 test_acc=0.6041666666666666\n",
      "Epoch 99 loss=0.030070148408412933 accuracy=1.0 val_acc=0.6857142857142857 test_acc=0.6046195652173914\n",
      "Epoch 100 loss=0.029325664043426514 accuracy=1.0 val_acc=0.6857142857142857 test_acc=0.6041666666666666\n",
      "Epoch 101 loss=0.028613392263650894 accuracy=1.0 val_acc=0.6857142857142857 test_acc=0.6046195652173914\n",
      "Epoch 102 loss=0.02793121710419655 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6041666666666666\n",
      "Epoch 103 loss=0.027277110144495964 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6041666666666666\n",
      "Epoch 104 loss=0.02665024995803833 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6041666666666666\n",
      "Epoch 105 loss=0.026048408821225166 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.603713768115942\n",
      "Epoch 106 loss=0.02547014318406582 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.603713768115942\n",
      "Epoch 107 loss=0.024914324283599854 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.603713768115942\n",
      "Epoch 108 loss=0.02437940239906311 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.603713768115942\n",
      "Epoch 109 loss=0.023864200338721275 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6041666666666666\n",
      "Epoch 110 loss=0.023367829620838165 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.603713768115942\n",
      "Epoch 111 loss=0.02288944460451603 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.603713768115942\n",
      "Epoch 112 loss=0.022427884861826897 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.603713768115942\n",
      "Epoch 113 loss=0.021982161328196526 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6041666666666666\n",
      "Epoch 114 loss=0.021551571786403656 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6041666666666666\n",
      "Epoch 115 loss=0.0211354810744524 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6032608695652174\n",
      "Epoch 116 loss=0.020733091980218887 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6028079710144928\n",
      "Epoch 117 loss=0.020343618467450142 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6019021739130435\n",
      "Epoch 118 loss=0.019966619089245796 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6014492753623188\n",
      "Epoch 119 loss=0.019601713865995407 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6014492753623188\n",
      "Epoch 120 loss=0.01924823224544525 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6009963768115942\n",
      "Epoch 121 loss=0.01890559121966362 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6009963768115942\n",
      "Epoch 122 loss=0.018573053181171417 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6009963768115942\n",
      "Epoch 123 loss=0.018250416964292526 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6000905797101449\n",
      "Epoch 124 loss=0.017937207594513893 accuracy=1.0 val_acc=0.6857142857142857 test_acc=0.6005434782608695\n",
      "Epoch 125 loss=0.017632879316806793 accuracy=1.0 val_acc=0.6857142857142857 test_acc=0.6005434782608695\n",
      "Epoch 126 loss=0.017337223514914513 accuracy=1.0 val_acc=0.6857142857142857 test_acc=0.6000905797101449\n",
      "Epoch 127 loss=0.017049791291356087 accuracy=1.0 val_acc=0.6857142857142857 test_acc=0.5996376811594203\n",
      "Epoch 128 loss=0.01677037589251995 accuracy=1.0 val_acc=0.6857142857142857 test_acc=0.5996376811594203\n",
      "Epoch 129 loss=0.016498537734150887 accuracy=1.0 val_acc=0.6857142857142857 test_acc=0.5996376811594203\n",
      "Epoch 130 loss=0.016234181821346283 accuracy=1.0 val_acc=0.6857142857142857 test_acc=0.5996376811594203\n",
      "Epoch 131 loss=0.01597682572901249 accuracy=1.0 val_acc=0.6857142857142857 test_acc=0.5996376811594203\n",
      "Epoch 132 loss=0.015726083889603615 accuracy=1.0 val_acc=0.6857142857142857 test_acc=0.6000905797101449\n",
      "Epoch 133 loss=0.015481753274798393 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6005434782608695\n",
      "Epoch 134 loss=0.01524385903030634 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6005434782608695\n",
      "Epoch 135 loss=0.015011751092970371 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6009963768115942\n",
      "Epoch 136 loss=0.014785613864660263 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6005434782608695\n",
      "Epoch 137 loss=0.014565042220056057 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6000905797101449\n",
      "Epoch 138 loss=0.014349889941513538 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6000905797101449\n",
      "Epoch 139 loss=0.014139854349195957 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6000905797101449\n",
      "Epoch 140 loss=0.013934796676039696 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6000905797101449\n",
      "Epoch 141 loss=0.013734581880271435 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6000905797101449\n",
      "Epoch 142 loss=0.013539088889956474 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6000905797101449\n",
      "Epoch 143 loss=0.013348042033612728 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6000905797101449\n",
      "Epoch 144 loss=0.01316142175346613 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.5996376811594203\n",
      "Epoch 145 loss=0.012978975661098957 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.5996376811594203\n",
      "Epoch 146 loss=0.01280051190406084 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.6000905797101449\n",
      "Epoch 147 loss=0.012626114301383495 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.6000905797101449\n",
      "Epoch 148 loss=0.012455460615456104 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6000905797101449\n",
      "Epoch 149 loss=0.012288510799407959 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6005434782608695\n",
      "Epoch 150 loss=0.012125165201723576 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6000905797101449\n",
      "Epoch 151 loss=0.011965337209403515 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6000905797101449\n",
      "Epoch 152 loss=0.011808915995061398 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6000905797101449\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 153 loss=0.011655798181891441 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6005434782608695\n",
      "Epoch 154 loss=0.011505785398185253 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6005434782608695\n",
      "Epoch 155 loss=0.011358906514942646 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6005434782608695\n",
      "Epoch 156 loss=0.011214920319616795 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6005434782608695\n",
      "Epoch 157 loss=0.011073877103626728 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6005434782608695\n",
      "Epoch 158 loss=0.010935710743069649 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6005434782608695\n",
      "Epoch 159 loss=0.010800261981785297 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6000905797101449\n",
      "Epoch 160 loss=0.010667545720934868 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6000905797101449\n",
      "Epoch 161 loss=0.01053738035261631 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6000905797101449\n",
      "Epoch 162 loss=0.010409791953861713 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6000905797101449\n",
      "Epoch 163 loss=0.010284650139510632 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6000905797101449\n",
      "Epoch 164 loss=0.010161920450627804 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6000905797101449\n",
      "Epoch 165 loss=0.010041462257504463 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6000905797101449\n",
      "Epoch 166 loss=0.009923247620463371 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6005434782608695\n",
      "Epoch 167 loss=0.009807263500988483 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6005434782608695\n",
      "Epoch 168 loss=0.009693419560790062 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6005434782608695\n",
      "Epoch 169 loss=0.009581669233739376 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6005434782608695\n",
      "Epoch 170 loss=0.009471924044191837 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6005434782608695\n",
      "Epoch 171 loss=0.009364166297018528 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6000905797101449\n",
      "Epoch 172 loss=0.00925836805254221 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6000905797101449\n",
      "Epoch 173 loss=0.009154461324214935 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6000905797101449\n",
      "Epoch 174 loss=0.009052390232682228 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.6000905797101449\n",
      "Epoch 175 loss=0.008952111005783081 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.5996376811594203\n",
      "Epoch 176 loss=0.008853581734001637 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.5996376811594203\n",
      "Epoch 177 loss=0.008756755851209164 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.5996376811594203\n",
      "Epoch 178 loss=0.008661641739308834 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.5996376811594203\n",
      "Epoch 179 loss=0.008568093180656433 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.5991847826086957\n",
      "Epoch 180 loss=0.008476126939058304 accuracy=1.0 val_acc=0.6885714285714286 test_acc=0.5991847826086957\n",
      "Epoch 181 loss=0.008385734632611275 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.5991847826086957\n",
      "Epoch 182 loss=0.008296912536025047 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.5991847826086957\n",
      "Epoch 183 loss=0.008209548890590668 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.5996376811594203\n",
      "Epoch 184 loss=0.008123602718114853 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.5996376811594203\n",
      "Epoch 185 loss=0.008039114996790886 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.5996376811594203\n",
      "Epoch 186 loss=0.007956001907587051 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.5996376811594203\n",
      "Epoch 187 loss=0.00787423737347126 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.5996376811594203\n",
      "Epoch 188 loss=0.0077937874011695385 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.5996376811594203\n",
      "Epoch 189 loss=0.00771462032571435 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.5996376811594203\n",
      "Epoch 190 loss=0.007636724505573511 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.5996376811594203\n",
      "Epoch 191 loss=0.0075600543059408665 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.5996376811594203\n",
      "Epoch 192 loss=0.007484584581106901 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.5996376811594203\n",
      "Epoch 193 loss=0.007410289254039526 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.5996376811594203\n",
      "Epoch 194 loss=0.0073371464386582375 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.5996376811594203\n",
      "Epoch 195 loss=0.007265167310833931 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.5996376811594203\n",
      "Epoch 196 loss=0.007194266188889742 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.5996376811594203\n",
      "Epoch 197 loss=0.007124484982341528 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.5996376811594203\n",
      "Epoch 198 loss=0.007055748254060745 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.5996376811594203\n",
      "Epoch 199 loss=0.006988039705902338 accuracy=1.0 val_acc=0.6914285714285714 test_acc=0.5996376811594203\n"
     ]
    }
   ],
   "source": [
    "def test(mask):\n",
    "    logits = model(graph)\n",
    "\n",
    "    test_mask_logits = tf.gather_nd(logits, tf.where(mask))\n",
    "    masked_labels = tf.gather_nd(labels, tf.where(mask))\n",
    "\n",
    "    ll = tf.math.equal(tf.math.argmax(masked_labels, -1), tf.math.argmax(test_mask_logits, -1))\n",
    "    accuarcy = tf.reduce_mean(tf.cast(ll, dtype=tf.float64))\n",
    "\n",
    "    return accuarcy\n",
    "\n",
    "model = GraphConvolutionModel()\n",
    "\n",
    "optimizer=tf.keras.optimizers.Adam(learning_rate=0.01, decay=5e-5)\n",
    "\n",
    "# 记录过程值，以便最后可视化\n",
    "train_loss_results = []\n",
    "train_accuracy_results = []\n",
    "train_val_results = []\n",
    "train_test_results = []\n",
    "\n",
    "num_epochs = 200\n",
    "\n",
    "for epoch in range(num_epochs):\n",
    "\n",
    "    loss_value, grads = grad(model, graph, labels, train_mask)\n",
    "    optimizer.apply_gradients(zip(grads, model.trainable_variables))\n",
    "\n",
    "    accuarcy = test(train_mask)\n",
    "    val_acc = test(val_mask)\n",
    "    test_acc = test(test_mask)\n",
    "\n",
    "    train_loss_results.append(loss_value)\n",
    "    train_accuracy_results.append(accuarcy)\n",
    "    train_val_results.append(val_acc)\n",
    "    train_test_results.append(test_acc)\n",
    "\n",
    "    print(\"Epoch {} loss={} accuracy={} val_acc={} test_acc={}\".format(epoch, loss_value, accuarcy, val_acc, test_acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到，经过200次迭代后，最终GCN网络在验证集上的准确率达到70%左右，在测试集中的Accuracy达到了60%左右。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  5.4 训练过程可视化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtQAAAILCAYAAADIRQsKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdd5hdVb3/8fd3WsqkJ5OQXggdgsDQpAsINooF6aJCRMVy8Xrvtf0s3Kter1iwoCiIINUGqCCIFAEhJBFSIRCSkN6TSZkk09bvj3MSJsMkOZMpZ8r79TzznLPX3nvt72xI8pk166wdKSUkSZIk7Z2CfBcgSZIkdWQGakmSJKkZDNSSJElSMxioJUmSpGYwUEuSJEnNYKCWJEmSmsFALUmtJCIKI2JTRIxqyWPbs4gYFxGb8l2HJLUlA7UkZWUD7favuojYUm/7kqb2l1KqTSn1SiktbMljmyoi/jsiUkR8okH7v2fbv5xjP4sj4tTdHZNSmpdS6tWMciWpwzFQS1JWNtD2ygbChcB76rXd0fD4iChq+yr32ivAhxq0XZZtbxEd7H5IUosxUEtSjrIjvfdExF0RsRG4NCKOj4jnImJ9RCyLiBsiojh7fFF2BHhMdvs32f0PRcTGiHg2IsY29djs/ndExCsRURERP4qIZyLiit2U/ywwICIOyJ7/FjL/BrzQ4Hs8JyKmZb+fpyPi0Gz7XcAw4KHsiP21ETE+W/OHI2Ih8Mj2tnr9DYyIW7P3Zl1E/D7bPjgiHsxeZ21E/GOv/8NIUp4ZqCWpac4H7gT6AvcANcBngEHACcDZwMd2c/7FwFeAAWRGwa9r6rERMRi4F/h89rrzgWNyqP124PLs+8uB2+rvjIijgV8AVwIDgVuA+yOiJKV0EbAUeEd2xP579U49GTgQeFcj17wTKAEOBoYAP8y2fx6YB5QB+2S/T0nqkAzUktQ0T6eU/pRSqkspbUkpTU4pTUop1aSU5gE3Aafs5vzfpZSmpJSqgTuAt+zFse8GXkwp3Z/d931gdQ613w5ckh1BvyDbZ30TgZ9mv6falNIt2faj99DvV1NKlSmlLfUbI2IkcDrw8ZTSupRSVUpp+0h0NZkR71HZ9idzqF+S2iUDtSQ1zaL6GxFxYET8JSKWR8QG4BtkRo13ZXm995XA7j7At6tjh9WvI6WUgMV7KjylNJ/MSPc3gVkppaUNDhkN/Gd2Gsb6iFgPDAWG76HrRbtoHwmsTilVNLLv28DrwN8j4rWI+Pye6pek9spALUlNkxps/xyYCYxPKfUB/h8QrVzDMmDE9o2ICPYcere7DfgcDaZ7ZC0Cvp5S6lfvq2dK6d7s/obfe6YxE+gbswgYFBF9GjlnQ0rp31JKY4DzyAT53Y3sS1K7ZaCWpObpDVQAmyPiIHY/f7ql/Bk4MiLek11Z4zNk5iLn4k7g7cDvG9l3E/DJiDg6Mnplr1Ga3b8CGJdrkSmlRcCjwE8iol9EFEfEyQDZfvfN/jBQAdRmvySpwzFQS1LzfI7McnQbyYxW39PaF0wprQA+CHwPWAPsS2a1jm05nFuZUno0pbS1kX2TgI8DNwLryCypd2m9Q74JfD07HeSzOZa7/fxXyATyT2W3DwAeAzYBzwA/TCk9nWOfktSuxK5/UydJ6ggiopDMChzvTyk9le96JKmrcYRakjqgiDg7IvpGRDcyS87VAM/nuSxJ6pIM1JLUMZ1IZh3n1WTWvj4vpbTHKR+SpJbnlA9JkiSpGRyhliRJkprBQC1JkiQ1g4FakiRJagYDtSRJktQMBmpJkiSpGQzUkiRJUjMYqCVJkqRmMFBLkiRJzWCgliRJkprBQC1JkiQ1g4FakiRJagYDtSRJktQMBmpJkiSpGQzUkiRJUjMYqCVJkqRmMFBLkiRJzWCgliRJkprBQC1JkiQ1g4FakiRJagYDtSRJktQMBmpJkiSpGQzUkiRJUjMYqCVJkqRmMFBLkiRJzWCgliRJkprBQC1JkiQ1g4FakiRJagYDtSRJktQMBmpJkiSpGQzUkiRJUjMYqCVJkqRmMFBLkiRJzWCgliRJkprBQC1JkiQ1g4FakiRJagYDtSRJktQMBmpJkiSpGQzUkiRJUjMYqCVJkqRmMFBLkiRJzWCgliRJkprBQC1JkiQ1Q7sK1BExMiIej4iXImJWRHwm3zVJkiRJuxMppXzXsENEDAWGppT+FRG9ganAeSml2bs6Z9CgQWnMmDFtVaIkSZK6qKlTp65OKZU1bC/KRzG7klJaBizLvt8YES8Bw4FdBuoxY8YwZcqUNqpQkiRJXVVEvN5Ye7ua8lFfRIwBjgAmNbJvYkRMiYgpq1atauvSJEmSpB3aZaCOiF7A74HPppQ2NNyfUroppVSeUiovK3vTqHurW7VxG1//0yz+OnM56zZXtfn1JUmS1H60qykfABFRTCZM35FS+kO+62nMqys3cuekhfzqmQUAHDCkN8eOG8CxYwdyzNgBlPXult8CJUmS1Gba24cSA/g1sDal9NlczikvL0/5mEO9raaWGYsrmDR/Lc/NW8PU19dRWVULwBGj+vGeCcN414ShDOnTvc1rkyRJUsuLiKkppfI3tbezQH0i8BQwA6jLNn8xpfTgrs7JV6BuqLq2jplLKvjna2v4y/RlzF62gQg4ZswA3n34MN5x6D4M6uXItSRJUkfVIQL13mgvgbqh11Zt4s/TlvGn6UuZu3IThQXBKfuXceHRIzntwMEUF7bL6euSJEnaBQN1nqSUmLNiIw+8uJTfTV3Myo3bKOvdjQ8cNYIPHj2S0QNL812iJEmScmCgbgdqaut4fM4q7pm8kMdeXkldguPHDeTS40Zz1iFDKHLUWpIkqd0yULczyyu28tspi7h78iKWrN/C8H49+PAJY/jg0SPp3b043+VJkiSpAQN1O1Vbl/jb7BXc8vR8nl+wll7divjg0SO54q1jGDmgZ77LkyRJUpaBugOYtmg9Nz89n7/MWEZKibMP3YePnjiOo0b3z3dpkiRJXZ6BugNZun4Lv352AXdNWsiGrTW8ZWQ/Jp48jrMP2YeCgsh3eZIkSV2SgboD2rytht9NXcwtz8zn9TWV7De4F9e8bTzvnjCMQoO1JElSmzJQd2C1dYm/zFjGj/7+Kq+u3MS4QaV88rTxnPuWYa4MIkmS1EYM1J1AXV3i4VnLueGxuby0bAOjBvTkmtPG894jhxusJUmSWpmBuhNJKfHoSyv50WOvMn1xBePKSvn3tx/AOw7dhwingkiSJLWGXQVqhzU7oIjgzIOHcP8nT+Cmy46iMIJP3PEvzvnxMzz16io6+g9JkiRJHYmBugOLCN5+yD789bMnc/0HDmft5iouu/l5Lv7FJF5YuC7f5UmSJHUJTvnoRLbV1HLXpIX86LG5rNlcxdsPHsLnzzqA/Yb0zndpkiRJHZ5zqLuQTdtquOXp+dz0j3lUVtXw3iNHcO2Z+zOsX498lyZJktRhGai7oLWbq7jxibn8+tnXCeDKk8Zy9Sn70rt7cb5LkyRJ6nAM1F3Y4nWVfPfhOdz34lIGlpbw2TP248JjRlHsUnuSJEk5c5WPLmxE/5784MIjeOCaE9hvSC++cv8szvr+P3hk1nJXBJEkSWomA3UXMmFEP+666jhu/lA5ETDx9qlcevMkXlmxMd+lSZIkdVgG6i4mIjj9oCE8/NmT+ca5hzBzyQbe8cOn+NoDs6iorM53eZIkSR2OgbqLKios4PLjx/DEv5/KRceM5LZnF3Da9U9w1/MLqa1zGogkSVKuDNRdXP/SEv77vMP486dOYvzgXnzhDzM458dPM2XB2nyXJkmS1CEYqAXAwcP6cM/E4/jRRUewdnMV7//Zs3z27hdYXrE136VJkiS1awZq7RARvOfwYfz9c6fwqbeN58GZy3nb9U/wsydfo7q2Lt/lSZIktUsGar1Jz5IiPvf2A/j7tadwwvhBfPuhl3nXDU8xad6afJcmSZLU7hiotUsjB/TkF5eX88vLy6msquWDNz3H5+6dxupN2/JdmiRJUrthoNYenXHwEP72b6fwydP25YFpSzj9+ie5Y9Lr1LkaiCRJkoFauelRUsjnzzqQhz5zEgcP7cOX/jiT82/8JzOXVOS7NEmSpLwyUKtJxg/uzZ1XHcsPPvgWlqzbwjk/fpqvPTCLDVt9KIwkSeqaWj1QR0SPiDgjIka39rXUNiKC844Yzt8/dwqXHjeaXz+7gNOvf5L7X1xCSk4DkSRJXUuLB+qIuDUiPpF9XwI8DzwCzImId7T09ZQ/fXsU841zD+WBT57I0L7d+czdL3LJLyfx2qpN+S5NkiSpzbTGCPVZwHPZ9+cAvYF9gK9lv9TJHDaiL3/8xAlcd96hzFhSwTt+8BTfe2QOW6tr812aJElSq2uNQN0fWJl9fzbw+5TSSuBu4OBWuJ7agcKC4LLjRvPY507lnYftww2PzeWsH/yDf7yyKt+lSZIktarWCNTLgUMjopDMaPWj2fZegJ9c6+TKenfjBxcewR1XHktBBJff8jyfuusFVm7wEeaSJKlzao1AfQtwDzATqAX+nm0/Fni5Fa6nduiE8YN46DMn8W9n7M/Ds5Zz+vVPcvuzC6h17WpJktTJtHigTil9A/gIcBNwYkqpKrurBvjflr6e2q/uxYV85oz9ePizJ3P4yH585f5ZvPenz7h2tSRJ6lSivS1zFhG3AO8GVqaUDt3T8eXl5WnKlCmtX5iaJaXEA9OWct2fX2Lt5m1c8daxXPv2/enVrSjfpUmSJOUkIqamlMobtrfGsnkXRMTb623/v4hYHBEPR8TQHLq4lcyHGdWJRATnviWzdvXFx47iV/+czxnXP8lDM5a5drUkSerQWmMO9de2v4mII4EvAjcAxcD1ezo5pfQPYG0r1KV2oG+PYv77vMP4w8ffyoDSEj5+x7/4yK2TWbS2Mt+lSZIk7ZXWCNSjgTnZ9+cD96WUvgNcC5zeEheIiIkRMSUipqxa5bJsHdERo/rzwDUn8OV3HcSk+Ws58/tP8tMn5lJdW5fv0iRJkpqkNQL1VjIPc4FMgN6+bF5FvfZmSSndlFIqTymVl5WVtUSXyoOiwgKuPGkcj157CqfuP5jv/HUO77rhKSYv8BcUkiSp42iNQP0UcH1EfAUoBx7Mtu8PLGqF66mDG9avBz+77Chu/lA5m7fV8oGfPct//m466zZX7flkSZKkPGuNQH0NUAW8H7g6pbQ02/4O4OFWuJ46idMPGsLfrj2Zq0/Zl9//azFvu/4JfjtlkR9alCRJ7Vp7XDbvLuBUYBCwAvhqSunmXR3vsnmd08vLN/DlP85kyuvrOGbsAL55/qGMH9wiM4YkSZL2yq6WzWu1QB0RbwMOBhIwO6X0eGtcx0DdedXVJX47dRHfeuhlNm+rYeLJ47jmtP3oUVKY79IkSVIXtKtA3eJP1YiI4cAfgaOA7dM9hkXEFOD8elNApN0qKAg+ePQozjhoCN988GV+8vhrPDBtKd8491BOO2BwvsuTJEkCWmcO9Q1ALTA+pTQypTQS2C/bdkMrXE+d3MBe3bj+gsO566rjKCks4MO/msxHb53MvFWb8l2aJElSy0/5iIgNwKkppX81aC8H/p5S6tuS13PKR9dSVVPHrf+czw1/n8vW6lo+9NYxfPr0/ejbozjfpUmSpE6uzR49vhs+sUPNVlJUwMST9+Xxfz+V9x81gluemc9p332COya9Tm1d+/qArSRJ6hpaI1D/HbghIkZub4iIUcAPgcda4Xrqgsp6d+Pb75vAn645kfGDe/GlP87kXTc8xT9fW53v0iRJUhfTGoH600BPYF5EvB4RC4DXgB7Ap1rheurCDh3el3smHsdPLzmSTdtquPgXk/jY7VNYuKYy36VJkqQuojWXzTsTOBAIYDYwF/hOSumClryOc6i13dbqWm5+ej4/eXwuNbWJy44fzSdPG8+A0pJ8lyZJkjqBNl+HupECDgf+lVJq0UWEDdRqaMWGrVz/yBx+N3UxpSVFTDx5HB89aSw9S1p8lUhJktSFtIcPJUptYkif7nzn/Yfz8GdP5vh9B3L9317hlP97gtufe53qWj8bK0mSWpaBWp3WfkN6c9Pl5fz+48czdmApX7lvJmd+70nue2GJK4JIkqQWY6BWp3fU6AHc87HjuOWKcroXF/LZe17kzO8brCVJUstosTnUEfHAHg7pA5zkHGrlU11d4pHZy/nBo6/y8vKNjCsr5dNv24/3HD6MwoLId3mSJKkda/UPJUbEr3I5LqX04Ra5YJaBWnujsWD9iVPHc87hwygp8hc3kiTpzfK+ykdrMVCrORoG66F9u/PRE8dy4TGj6NXNVUEkSdIbDNTSbqSUePKVVfz8yXk8O28NvbsXcdlxo7nihDEM7t093+VJkqR2wEAt5WjaovX8/B+v8dDM5RQXFHDeEcO4/PgxHDq8b75LkyRJeWSglppowerN/OKpefzhX0vYUl3LUaP786G3juHsQ/ZxnrUkSV2QgVraSxVbqvnd1MXc/uwCFqyppKx3Ny4+ZhQXHTOKffo6HUSSpK7CQC01U11d4slXV3HbPxfw+JxVFAScsn8ZF5SP5PSDhjhqLUlSJ2egllrQgtWb+e3URfxu6mJWbNjGgNISznvLcC44egQH7tMn3+VJkqRWYKCWWkFtXeIfr67id1MW88js5VTXJg4Z1of3HD6Md08Yyoj+PfNdoiRJaiEGaqmVrd1cxf0vLuG+F5cybdF6AI4c1Y93TxjGuyYMZUgf51tLktSRGailNrRwTSV/mr6UP09fxkvLNhABR48ZwNsPHsIZBw1hzKDSfJcoSZKayEAt5cnclRv507RlPDxrOS8v3wjA+MG9OP2gwZx50BCOGNWfwoLIc5WSJGlPDNRSO7BobSWPvrSCR19awaR5a6mpS/TvWcwJ4wdx4vhBnLjfIOddS5LUThmopXamYks1/3hlFY+/vJKn565m5cZtAIwdVMqJ4wdxwvhBHDN2AANKS/JcqSRJAgO11K6llHh15SaeenU1z8xdzXPz1lBZVQtkpoccPaY/5aMHcMzYAYzo34MIp4hIktTWDNRSB1JVU8e0xeuZvGAtUxasY8qCtWzYWgPAkD7dOHxEPw4f2Y8JI/py2PC+9OvpKLYkSa1tV4G6KB/FSNq9kqICjh4zgKPHDAAyT2l8ZeVGJs9fy5TX1zF9cQWPzF6x4/jRA3syYUQ/Dh7ahwOH9ubAfXqzT5/ujmRLktQGHKGWOqiKympmLq1g2uL1TF9UwYwlFSxZv2XH/r49ijlwn0y4Hj+kN+MGlTKurNSgLUnSXnKEWupk+mZXBzlh/KAdbRVbqpmzfCMvL9/Ay8s38vKyDfxu6mI2Z+djA/QoLmRMNlyPG1TK2EGljCvrxegBPenXs9iwLUlSExmopU6kb49ijhmb+fDidikllm/YyvxVm3lt9Wbmr9rM/NWbmLmkgodmLKOu3i+pepYUMqxfD4b368Hw/pnXEf177Ggb0qe7a2ZLktSAgVrq5CKCoX17MLRvD95abzQbMh9+XLi2knmrNrFwbSVL1m9h6fotLFm/hemL17Ousnqn4wsCBvbqxuDe27+6U9a7G4P7ZLbLendncO9u9OtZTK9uRY52S5K6BAO11IWVFBUwfnAvxg/u1ej+zdtqWFaxhcXrMiF7ecVWVm7YxqpN21i5cSuzlm5g9aZtO41yb1dcGPTrWUL/nsU7XgeUljRoy7zv0yMTwHt3L6K0pIgCR8ElSR2IgVrSLpV2K2L84N6MH9x7l8fU1iXWbq5i5catrNy4jdUbt7G+spq1lVWsr6xi3eZq1lVWMX/1Zv61cD3rK6uort39h6G3h+sdr92L6V2vrWe3InqWFNKjuJAe2deeJfXfF+3Y17OkkO7FhU5VkSS1mnYXqCPibOCHQCHwy5TSt/NckqTdKCwIynp3o6x3Nw7J4fiUEpu21bC+MhO011VWs3FrNZu21rBxaw0bt9Xs2N60LdNWsaWaxesqdxyzpbp2zxdqoFtRwY7AXVJUQLeiguxrISWFBW9uy26/0bbzvpLCAooKg+LCAooKsq+FQVFBAcWFQdGb2jNtxdnXosKguOCNfU6PkaSOq10F6ogoBH4CnAksBiZHxAMppdn5rUxSS4kIencvpnf3YkYO6LlXfdTVJbbW1FJZVcuWqlq2VGdeK6tq2VJdw5aqOiqrathavb3tjf1bq2upqq1jW3Vd5rWmlqqaOiora9hWU0dVTR3bsl9VNbWZ19o6WnuF0Uzgrheys4G8IILCgsxXQZB9rd+WfR9BQQEUFRRQUBAUNji2IHvMG20N9mdfi+odu+M1oKAgiIAgux3Z7XhjuyC7HQ23yW4XbD+vXlsj5zTW907XLmik7x3nvLFdv+/gjeO397X9Z5io1xcN9gdAg+2G/RDkdI1MV7voxx+opA6tXQVq4BhgbkppHkBE3A2cCxioJe1QUBD0LCmiZ0nb/BWWUqK6NlFVuz1wZ0J4dW2ipq6OmtpEdW0dNXXZ12x7dW1q8D5zTM2OYzPvq3dqe+P8mtpEbUrU1SVqU+YHidq6+m2Z7brtr3VQWVOz07Hb99U/p66OxvupS9TU769jP6agQ9ptaOfNYX97EH8j+O/6hwh2Om8X/US9a9aracc1Gl5zpze7Pqb+DxT1t3f+3ndxzk7973z+GzU0fm5Tamr8+9jFdXdT056+n1xqopE+96amht/a3tT05o3G//9o5LDGa2ps324u1lj/BRFcd96htCftLVAPBxbV214MHNvwoIiYCEwEGDVqVNtUJqnLighKioKSogLolu9q2k5KmVC9/bUuJVKCRL3tup23dxyzm+3tfSaybXWZV7Zv77hWyp73xrk02N5RT90b57JTPezoJ/FGLYk3aqDevuzmzsdnbsYb7fXfZ/uhkXNSg36339PG9r/pGo30Q/26t5/TyHXSbvphp3uw62vUr3f7PWFHH/W33zg6NXiTvbv16tq5vfH+du5kp3u3FzW9+TpNqyml7R2nN53blJpo5D7tuab69e/8Pe3q3N3VRGp8/+5qath3Y9u7rHE35+yq/4Zd73zeGxthoN6jxn7n9ab/dCmlm4CbIPOkxNYuSpK6oojM1JHG/2qWJG1XkO8CGlgMjKy3PQJYmqdaJEmSpD1qb4F6MrBfRIyNiBLgQuCBPNckSZIk7VI0NtconyLincAPyCybd0tK6X/2cPwq4PW2qK0Rg4DVebp2R+T9ahrvV9N5z5rG+9V03rOm8X41nfesadr6fo1OKZU1bGx3gbojiYgpKaXyfNfRUXi/msb71XTes6bxfjWd96xpvF9N5z1rmvZyv9rblA9JkiSpQzFQS5IkSc1goG6em/JdQAfj/Woa71fTec+axvvVdN6zpvF+NZ33rGnaxf1yDrUkSZLUDI5QS5IkSc1goJYkSZKawUAtSZIkNYOBWpIkSWoGA7UkSZLUDAZqSZIkqRkM1JIkSVIzGKglSZKkZjBQS5IkSc1goJYkSZKawUAtSZIkNYOBWpIkSWoGA7UkSZLUDEX5LqC5Bg0alMaMGZPvMiRJktTJTZ06dXVKqaxhe4cP1GPGjGHKlCn5LkOSJEmdXES83lh7m035iIhbImJlRMzcxf6IiBsiYm5ETI+II9uqNkmSJGlvteUI9a3Aj4HbdrH/HcB+2a9jgRuzr1Kr21pdy6ZtNfkuQ5Ik5WBQr275LmEnbRaoU0r/iIgxuznkXOC2lFICnouIfhExNKW0rE0KVJdUW5f49T8XcP0jc9hcVZvvciRJ0h4UBMz71rvyXcZO2tMc6uHAonrbi7NtbwrUETERmAgwatSoNilOnc/MJRV88Y8zmL64glMPKOP0AwfnuyRJkrQnEfmu4E3aU6Bu7O6kxg5MKd0E3ARQXl7e6DFSY+Ys38iDM5bx8KzlvLx8I4N6lfCji47g3ROGEu3wD6gkSWr/2lOgXgyMrLc9Aliap1rUyayvrOKbD77EvVMWEwFHjx7Al991EB84aiR9exbnuzxJktSBtadA/QBwTUTcTebDiBXOn+66tlbX8sLC9WSm1DfPwrWV/N/Dc1i/pZqrT9mXK08a2+4+zCBJkjquNgvUEXEXcCowKCIWA18FigFSSj8DHgTeCcwFKoEPt1Vtal/WbNrG5bc8z6ylG1qsz7eM7Mdv3nsYBw3t02J9SpIkQduu8nHRHvYn4JNtVI7aqeUVW7n05kksWlvJdz9wOCP792h2n8VFBRw+oh+FBc6RliRJLa89TflQF7dobSUX//I51m2u5tcfOYbjxg3Md0mSJEl7ZKBWu/Hl+2ayvrKaO648lsNH9st3OZIkSTlps0ePS7szZ/lGnnxlFR87eZxhWpIkdSgGarULv3hqHj2KC7nk2NH5LkWSJKlJDNTKu5UbtnL/i0u4oHwE/UtL8l2OJElSkxiolXe3/nMBtXWJj5w4Nt+lSJIkNZmBWnm1eVsNv3nudc4+dB9GDyzNdzmSJElNZqBWXt0zeREbttZw1Unj8l2KJEnSXskpUEfEDyLi0NYuRl3L6k3buOGxVzlu3ACOGNU/3+VIkiTtlVxHqI8GpkXE8xExMSJ8frOa7b//PJvKbbX893n+rCZJkjqunAJ1SukE4GDgceCrwNKIuC0iTmnN4tR5PfnKKu57cSmfOG1fxg/une9yJEmS9lrOc6hTSnNSSv8JjAQuBHoBj0TEqxHxXxExoLWKVOdSWVXDl/44g33LSvn4qfvmuxxJkqRm2ZtHjxcDfYC+QCGwELgM+HJETEwp3dmC9amTuPGJ1/jb7OUArN9SzeJ1W7j3Y8fTragwz5VJkiQ1T84j1BFRHhE/BZYB3wGeA/ZLKZ2eUjoE+BLw/dYpUx3Z8oqtfO9vc6jYUk1ptyKG9+vBV99zMMeM9ZcakiSp48tphDoiZgAHAA8DVwB/SSnVNjjsTgzUasT2B7f86opjGDWwZ77LkSRJalG5Tvm4F7glpbRkVweklFbhutZqYNO2Gu6Y9DrvOHSoYVqSJHVKuQbq/6WRsBwR3YG6lFJVi1alTuPeyYvYuLWGK0/yseKSJKlzynVE+bfAJxppv5rM6LX0JjW1ddz89HyOHtPfB7dIkqROK9dAfQLwSCPtf85sGfMAACAASURBVAPe2nLlqKNLKVFbl/l6cOZylqzf4mPFJUlSp5brlI+eQE0j7XWAT+UQAIvXVfKRWyfzyopNO9rGDirljIOG5LEqSZKk1pVroJ4OXETmKYn1XQzMzPViEXE28EMy61f/MqX07Qb7RwG/Bvplj/mvlNKDufav/Hlt1SYu/eUkNm+r4dOn70dRQQBw2gGDKci+lyRJ6oxyDdTXAfdFxHjgsWzb6cAHgPNz6SAiCoGfAGcCi4HJEfFASml2vcO+DNybUroxIg4GHgTG5Fij8mT20g1cfsskUoK7Jx7PwcP65LskSZKkNpPTHOqU0l+A9wCjgRuyX6OAc1JKf87xWscAc1NK87KrgtwNnNvwUmSewgiZJzEuzbFv5UFtXeKWp+fzgZ/9k+LCAu692jAtSZK6npwfPZ5S+ivw12ZcaziwqN72YuDYBsd8DXgkIj4FlAJnNNZRREwEJgKMGjWqGSVpb81euoEv/GE60xZXcMr+ZXzrvYcxrF+PfJclSZLU5tryQSyNTaRNDbYvAm5NKY0A3gncHhFvqjGldFNKqTylVF5WVtYKpWp3Xl+zmffd+E+WrN/CDRcdwa0fPtowLUmSuqxcHz1eAnyJTOAdBRTX359SKsyhm8XAyHrbI3jzlI6PAmdn+3w2++CYQcDKXOpU60sp8aU/zqSoIPjTp05kaF+DtCRJ6tpyHaG+DvgQcD2ZpfI+T+YDhmto/IEvjZkM7BcRY7MB/ULggQbHLCTzYUci4iCgO7Aqx/7VBv74whKenrua/3jHgYZpSZIkcg/UFwBXp5R+DtQC96eUPk1mGb0zc+kgpVQDXAM8DLxEZjWPWRHxjYg4J3vY54CrImIacBdwRUqp4bQQ5cmaTdu47s+zOWp0fy45xrnrkiRJkPuHEocA25e320RmnWjIfEjxf3O9WHZN6QcbtP2/eu9nk3kqo9qh//nLS2zaVsO33nuYa0tLkiRl5TpCvRAYln0/Fzgr+/54YEtLF6X25/4Xl/CHF5Zw9Sn7sv8QH44pSZK0Xa6B+o9k5zaTedLh1yNiPnAr8MtWqEvtyKylFfzn76dz7NgBfPr0/fJdjiRJUruS05SPlNIX6r3/XUQsIjM145UmPNhFHdD6yiqu/s1U+vUo4ccXH0lxYVuutChJktT+7TFQR0Qx8Bvgiyml1wBSSpOASa1cm/Ksri7x6btfZEXFNu752HGU9e6W75IkSZLanT0ON6aUqoG38+aHsKiT+9P0pfzjlVV85T0Hc8So/vkuR5IkqV3K9ff3fwDe25qFqH2prq3je397hQP36e0SeZIkSbuR67J5C4EvR8RJwBRgc/2dKaXvtXRhyq97pyzi9TWV3HJFuUvkSZIk7UaugfoKYB0wIftVXwIM1J3Ilqpafvjoq5SP7s9pBwzOdzmSJEntWq6rfIxt7ULUfvz62QWs3LiNH198JBGOTkuSJO2Oa6BpJy8t28CNT7zGqQeUcczYAfkuR5Ikqd3LaYQ6Im7Y3f6U0qdbphzly5aqWn7491f5xVPz6NejmC++86B8lyRJktQh5DqH+rAG28XAgdnz/9WiFanNpJR4ZcUmHp61nN9OXcSitVv4YPlIvvDOA+nXsyTf5UmSJHUIuc6hPq1hW0R0B24GnmrpotT6pixYy3/8bjrzVm8mAo4a1Z/vvO9wjt93YL5LkyRJ6lByHaF+k5TS1oj4H+Bh4GctV5Ja21OvrmLibVMZ0qcb/3P+oZx58BAG9+6e77IkSZI6pL0O1FllQK+WKERt4+FZy/nUnS8wrqyU2z96rI8TlyRJaqZcP5R4bcMmYChwCfBgSxellrVhazWPv7ySR2at4K+zlnPY8L7c+uGjnSctSZLUAnIdof5Ug+06YBXwK+BbLVqR9mjB6s08PGs5j8xewbrNVZx6wGDOOmQIIwb05NHZK3hk9nKmL67IPHIHqKyupbYuMahXNy45dhT/cfaB9OrW3F9OSJIkCXywS4eSUuILf5jB3ZMXAXDIsD6MHNCT3zz3Orc8M3/HcfuWlXLuW4ZRUlgIQGm3Qk7Zv4wjRvWn0MeIS5Iktahcp3yUAAUppa0N2rsDdSmlqtYoTjv71TMLuHvyIj50/GiuPGkcIwf0BGDTthqemLOS5RVbOfWAMsYP7p3nSiVJkrqOXJ+U+FvgE420Xw3cm+vFIuLsiJgTEXMj4r92ccwFETE7ImZFxJ259t3ZPTdvDf/z4EucefAQvvqeQ3aEaYBe3Yp494RhXHnSOMO0JElSG8t1Iu0JwJcaaf8b8MVcOoiIQuAnwJnAYmByRDyQUppd75j9gC8AJ6SU1kXE4Bzr6xS21dSydvObB/srtlRzzZ3/YvSAnnzvgsMpcNqGJElSu5FroO4J1DTSXgfkOiR6DDA3pTQPICLuBs4FZtc75irgJymldQAppZU59t3hzV+9mUt/OYkl67c0ur+0pJC7Jx5H7+7FbVyZJEmSdifXQD0duAj4aoP2i4GZOfYxHFhUb3sxcGyDY/YHiIhngELgaymlv+bYf4f18vINXPrL56lLievOPYTiwjfPxDlydH+nc0iSJLVDuQbq64D7ImI88Fi27XTgA8D5OfbR2DyF1Eg9+wGnAiOApyLi0JTS+p06ipgITAQYNWpUjpdvn15ctJ4P3fI8PYoL+c2VxxqaJUmSOpicPpSYUvoL8B5gNHBD9msUcE5K6c85XmsxMLLe9ghgaSPH3J9Sqk4pzQfmkAnYDeu5KaVUnlIqLysry/Hy7c/razZz+c2T6NujmN9efbxhWpIkqQPK+eke2akXzZl+MRnYLyLGAkuAC8lMGanvPjJTS26NiEFkpoDMa8Y1263Kqho+dvtUIoLffPTYnVbtkCRJUseR0wh1RJwSEafsov3kXPpIKdUA1wAPAy8B96aUZkXENyLinOxhDwNrImI28Djw+ZTSmlz670hSSvzn72cwZ8VGbrjoCEYNNExLkiR1VLmOUH8f+EYj7X2ArwFH5dJJSulB4MEGbf+v3vsEXJv96pS2Vtdy0z/m8adpS/n8WQdwyv4dd8qKJEmScg/UBwDTGmmfkd2nPXjs5RX8fuoSnpizks1Vtbzj0H34xKn75rssSZIkNVOugXoLMAyY36B9BOBjx/fgoRnL+Pgd/2JQr26c85bhnHXIEE4cP4gIH9AiSZLU0eUaqB8Gvh0R52x/6EpEDAC+md2nXXh1xUb+/bfTOGJUP+6eeBzdigrzXZIkSZJaUK6B+t+BfwALImJ6tm0CsIrMah1qxIat1Xzs9qn0KCnixkuOMkxLkiR1QrmuQ70MOJxMsJ5OZu7054DDgINbrboOrK4uce0901i4tpKfXnIk+/Ttnu+SJEmS1Aqasg51JfALgIgYDnwYmEXmYS8OvTbw48fn8uhLK/jqew7mmLED8l2OJEmSWklOI9QAEVEYEedHxF+ABWQeOf4zYHwr1dZhPf7ySr7/6Cucf8RwrnjrmHyXI0mSpFa0xxHqiDgAuBK4HNgM3AmcBVyWUprduuV1PAtWb+Yzd7/AQfv04ZvnH+ZKHpIkSZ3cbkeoI+Ip4DmgH3BBSmlcSunLQGqL4jqaRWsr+djtUykoCH5+2VH0KHEmjCRJUme3pxHq44GfAL9IKc1sg3o6pOraOm55ej7ff/QVCiP4+WXljBzg48QlSZK6gj0F6nLgKuCpiFgA3Abc1dpFdSSbttVw4U3PMnPJBs48eAhfP+cQhvXrke+yJEmS1EZ2O+UjpfRiSumTwFDge8C5wKLsee+KiP6tX2L79sSclcxcsoHvvH8Cv7i83DAtSZLUxeS6DvXWlNLtKaVTgYOA/wP+DVgeEQ+1Yn3t3jNzV9O7WxHvPWJ4vkuRJElSHuS8bN52KaW5KaX/AkYCFwBVLV5VB/LM3DUcO24gRYVNvpWSJEnqBPY6BaaUalNK96eUzm3JgjqSRWsrWbi2khPHD8x3KZIkScoTh1Wb4Zm5qwE4YfygPFciSZKkfDFQN8PTc1czuHc3xg/ule9SJEmSlCcG6r1UV5d49rU1nDB+kE9DlCRJ6sIM1Hvp5eUbWbO5yukekiRJXZyBei+9MX/aDyRKkiR1ZW0aqCPi7IiYExFzI+K/dnPc+yMiRUR5W9bXFM+8tppxZaUM7euDXCRJkrqyNgvUEVEI/AR4B3AwcFFEHNzIcb2BTwOT2qq2pqqqqWPSvLWcsK/TPSRJkrq6thyhPgaYm1Kal1KqAu4m8yjzhq4DvgNsbcPamuTFRevZUl3r/GlJkiS1aaAeDiyqt70427ZDRBwBjEwp/bkN62qy8YN78Z33TeD4fZ0/LUmS1NUVteG1GltbLu3YGVEAfB+4Yo8dRUwEJgKMGjWqhcrL3YDSEi44emSbX1eSJEntT1uOUC8G6qfQEcDSetu9gUOBJyJiAXAc8EBjH0xMKd2UUipPKZWXlZW1YsmSJEnS7rVloJ4M7BcRYyOiBLgQeGD7zpRSRUppUEppTEppDPAccE5KaUob1ihJkiQ1SZtN+Ugp1UTENcDDQCFwS0ppVkR8A5iSUnpg9z00burUqasj4vWWrLUJBgGr83Ttjsj71TTer6bznjWN96vpvGdN4/1qOu9Z07T1/RrdWGOklBprVw4iYkpKqd2uld3eeL+axvvVdN6zpvF+NZ33rGm8X03nPWua9nK/fFKiJEmS1AwGakmSJKkZDNTNc1O+C+hgvF9N4/1qOu9Z03i/ms571jTer6bznjVNu7hfzqGWJEmSmsERakmSJKkZDNSSJElSMxioJUmSpGYwUEuSJEnNYKCWJEmSmsFALUmSJDWDgVqSJElqBgO1JEmS1AwGakmSJKkZDNSSJElSMxioJUmSpGYwUEuSJEnNYKCWJEmSmqEo3wU016BBg9KYMWPyXYYkSZI6ualTp65OKZU1bO/wgXrMmDFMmTIl32VIkiSpk4uI1xtrd8qHJEmS1AwdfoRa6mq2VNUye9kGZixez0vLNrKtphaAiGDMwFImjOjLYSP6MqhXtzxXKklS12CgltqBTEiuYPriCmYsrmD6kgpeX7OZfct6cdjwvowr68X81ZuYvriCV1duorYuATCwtIRe3TN/jGtqE/e9uISU2cWwvt05bERfJozox4DSkkavO6RPNw4b3o+y3oZvSZL2loFaypMVG7Zyz+RFPDhjGa+s2Eg2I+8IuafuX8bcVZt47OWV/HbqYgaUlnDY8L6cefAQDhueCcpD+nQjInb0uWlbDbOWVDBjSTacL6ng4Vkr9ljL0L7dGdq3+0591d934dGjeOu+AykoePN+SZK6ukjbh7M6qPLy8uSHEtUR1NTW8erKTcxYXMFjL6/kby+toLYucfy4gRw9pj8TRvTjsBF9GdKn+07npZTYsKWGPj2KGg28e7JxazWbt9W+qT2RWLxuC9MWrWfGkgrWbKpq9JjZSzewrrKasYNKOf+I4Rwxqh+HDe9Lv56Nj3pLktRZRcTUlFJ5w3ZHqKUWUFNbx99fXsmdkxYye9mGRo/ZsKWabTV1QGaqxpUnjeXiY0YxemDpbvuOCPr2LN7r2np3L6Z398bPH9q3B0ePGbDb87dW1/LXmcu5Y9LrfO9vr+xoHzWgZ2ZKyfC+jB/ci8K9HL2uqqljzvKNTF9SwUvLNuy4R22hd7ciDh3elwkj+jKurJSC7A8sPYoLOWhYH/rs4r51Zqs2buOVFRuprm27/w7t0agBPRkzsLTNfiuTUuYH3PmrN1OX54GuBCxdv4WZ2d90rdy4rdHjtv/5OWx4X/Yd/MafH+XPqo3bdvyGcsn6Lfkup9UUBEz64hn5LmMnjlBLe6GuLjFv9WZmLFnPtEUV/HXmcpZv2Mo+fbpz8v6DKCx48wI6vboV7vjHpy3/oW5JFZXVzFy6fTpJ5ntvqb+0x5WVcsiwvvTq1nY/56/ZtI2ZSypYWrG18ZoGlbL/kN4UFeb+36qwIDJz30f05dBhfendffffT11KzFu1mRnZqTobtlQ36XtoKds/7LpsF/eiK+rdvYhDh/VlYK/W/W1MxZZqZi6pYF1lfv7b70rfHsVMGNGXEf17AG/+M7CnPz/Kj9KSzL81YweV7tVvNTuCCPjm+Yfl6dqNj1AbqKVG1NUlFqzJhJwNW2syjSmxKDtFYtbSDWzalmnvUVzIseMGcPExo3jbgYMpKuxaq1Gu2bSNhWsr2du/SQojGFtWmtfR4FUbt7FoXeWO7Yot1czKjvLMa+KIYVVNHUvWb2Fv/mrt072IQXn6gGhxQQEHDu3NYcP7cvDQPnQvKcxLHe1BXV3mh5xpi9czc0kFG7N/1ltLj+JCDh2WWZ2nqT/AtZZBpd0YOaBHToGs4Z8f5U+f7sWMG9QxB2w6CgO1urzXVm3iWw++xBkHDeGC8pE7/sJ5efkGvvvwK6zamBllqUuwYPXmRv8RLSkq4OChfTJL02U/GLhvWWmXC9HavY1bq5m1dAOzl+Y2hWVE/x5MGNGXUQN6dtoRJUnqDAzU6tIen7OST9/1AluqaqmpSxw5qh9fetdB/HXmcm55ZgF9uhdx+Mh+O44f0b8HE4b349DhfXdaUq5vj2JKigzPkiR1RX4oUV3Syg1b+e3UxXz3kTkcuE8fbrrsKCbNX8u3HnyJ9934LAAXHTOS/zjrQPrvYq1mSZKk3TFQq9N5fc1mvv3Qy/xr4TpWbMh8Ov1dhw3l/z4wgZ4lRYwc0JMzDhrM7c++zlvHD+Ko0f3zXLEkSerIDNTqVJ56dRXX3PkCKSVOPyjzAJTDR/bjyFH9dpqb2q9nCZ86fb88VipJkjoLA7U6rC1Vtby+djN12c98PfXqKv73ry+z/5De/OLyckYO6JnfAiVJUpdgoFaH8tqqTdz05DymLV7Pqys3UVu384dqzzpkCN+74C2UtuFaxpIkqWszdajD+PtLK/js3S+SgKNG9+fMg4ew/5DeFGeXrOvVrYi37jvQ9TclSVKbMlCr3Usp8dMnXuO7j8zhkGF9+Pll5Qzv1yPfZUmSJAEGarVzW6pq+Y/fT+dP05bynsOH8Z33TaBHF36CmyRJan8M1Gq3lqzfwsTbpjB72Qb+8+wDufqUcT5FTpIktTsGarVLT76yimvveZGqmjpu/lA5bztwSL5LkiRJapSBWu3K8oqtXPeX2fxl+jL2LSvl55eVM35wr3yXJUmStEsGarUb97+4hC/+YQbVdYlrz9yfiSePo3ux86UlSVL7llOgjoivA0tTSj9v0D4RGJpS+nprFKeu448vLObae6dx9OgB/N8HJjB6YGm+S5IkScpJQY7HXQFMa6T9ReDDuV4sIs6OiDkRMTci/msXx1wQEbMjYlZE3Jlr3+q4Hpi2lM/dO43jxg7k1x85xjAtSZI6lFynfAwBVjTSvgrYJ5cOIqIQ+AlwJrAYmBwRD6SUZtc7Zj/gC8AJKaV1ETE4x/rUAaWUuHfKIr74x5mUjxnAzVeUuySeJEnqcHIN1AuBE4H5DdpPApbk2McxwNyU0jyAiLgbOBeYXe+Yq4CfpJTWAaSUVubYtzqYV1ds5Mv3zWTS/LUcN24Av/zQ0fQscUq/JEnqeHJNML8Evh8RRcBj2bbTge8A382xj+HAonrbi4FjGxyzP0BEPAMUAl9LKf21YUfZudsTAUaNGpXj5dVe3Pz0fL714EuUdivim+cfxoVHj/Rx4ZIkqcPKNVD/H1AG3AgUZ9uqgR+nlL6dYx+NJabUSD37AacCI4CnIuLQlNL6nU5K6SbgJoDy8vKGfagde2jGMq7782zOPHgI33rvYQzq1S3fJUmSJDVLToE6pZSAz0fEdcAhZMLxzJTShiZcazEwst72CGBpI8c8l1KqBuZHxBwyAXtyE66jdmr20g1ce+80jhjVjx9ffATdipwvLUmSOr6cVvmIiLKIGJZS2pBSejal9M+U0oaIGBYRZTleazKwX0SMjYgS4ELggQbH3Aeclr3mIDJTQObl2L/asTWbtnHVbVPo26OYn196lGFakiR1GrlO+bgD+D3w8wbt7wLeD5y1pw5SSjURcQ3wMJn50beklGZFxDeAKSmlB7L73h4Rs4Fa4PMppTU51qh2ZuaSCv42ewXTF6/nxUXrqayq5bdXH8/gPt3zXZokSVKLicxsjj0cFLEOOD6l9HKD9v3JTNEY0Er17VF5eXmaMmVKvi6vXXjylVVc9esp1NTVsd/g3hw2oi/vO3IEx+87MN+lSZIk7ZWImJpSKm/YnusIdTFvfBixvm7ZL2mHp19dzcTbpjB+cC9u/+gxDPSDh5IkqRPL9UmJzwMfa6T948DUlitHHd2zr63hytsmM3ZQKb+58ljDtCRJ6vRyHaH+CvBoREwA/p5tOx04msyTDyWen7+Wj9w6mZH9e/KbK49lQGlJvkuSJElqdTmNUKeUngFOILPM3cXAJdn3J6SUnm698tRRTH19LVf86nmG9uvOHVcd6/rSkiSpy8j5Wc8ppX+RWepuJxFRmlLa3KJVqUN5YeE6PnTLZIb06c5dVx3H4N6u4iFJkrqOXOdQv0lEHBcRNwPLWrAedTB/m72CS385iQGlJdx51bEMcUk8SZLUxTQpUEfEwIj4t4iYCTxJ5smHn2uVytSupZT48WOvMvH2Kew7uBf3fux4hvbtke+yJEmS2lxOUz4i4u3AVcC7gWnAgcBbU0rPt2JtaqdWb9rGV+6byUMzl3PeW4bx7fdNoHuxTz6UJEld024DdUR8FfgwkIDfAF9MKb0aEdXApjaoT+1IbV3irucX8p2/vsyW6lq++M4DueqkcUREvkuTJEnKmz2NUH8F+Bbw1ZRSXRvUo3Zq7eYqPvrrybywcD3HjxvIdecdyvjBvfJdliRJUt7tKVB/kcwI9Ycj4i7gtpTSjNYvS+3Jus1VXPLLScxbtYnvf/BwznvLcEelJUmSsnb7ocSU0ndSSgcBFwFlwLMRMQ0IYFAb1Kc8q6is5tKbJ/Haqk384vJyzj9ihGFakiSpnlwf7PJUSukKYChwI/AC8HhEPBcRrvLRSa3etI3LbpnEqys28fPLjuLk/cvyXZIkSVK706Rl81JKG1NKP0spHQ0cAUwGvtAqlSmvZi6p4JwfPc0rKzZy46VHctoBg/NdkiRJUru01w92SSlNTyl9ChjWgvWoHXhg2lLe/7N/AvC7q9/K6QcNyXNFkiRJ7VfOjx7flZRSVUsUovbh9mcX8JX7Z3H0mP789JKjKOvdLd8lSZIktWvNDtTqPO6ctJCv3D+LMw4azE8vOYqSor3+BYYkSVKXYWISAPdOXsQX/ziD0w4o4yeXHGmYliRJypGpSfzztdX85x+mc9J+g7jx0qPoVuRjxCVJknJloO7ittXU8uX7ZjKyf09uuqyc7sWGaUmSpKbY5RzqiPj/7d15fFx1vf/x12cmk7Vp05LuCy1QkLKXQBEsilyugEBBlE0UEChXBfV68YqXq1fL9V70p+ICVwUBQdllsSCCICKy2abI0hZa0n2jaZu0TbPO8vn9MZMwTSfNTCeZmTTv5+ORRzPfOTnnM9+cTN/nzPd8TyPg6azE3Uf0WUWSU7e9sJzlm5r59WXHUFasMC0iIiKSqd1dlHhtzqqQvFi9pYWfPVfH6YeN4SOaZ1pERERkj/QYqN399lwWIrmxvS1MJBr/4OFbcxdSFDC+dcYhea5KREREZODStHmDxHvb2rjhicX84a0NO7V/84xpjBlWmqeqRERERAa+tAK1mYWA64ALgUlAcfLz7l6c6uck/zoiMe5+ZSU3PbOUSMy56sP7MW5YGQAjKoo5/bCx+S1QREREZIBL9wz1HODTwPeAHwDXA1OATwHf6p/SJBtrGlq4d95qHqpdw+YdHZx00Ei+c9ahTNqnPN+liYiIiOxV0g3U5wNXufsfzexG4BF3X2Zmi4GTgJ/3W4WSkS072vnfP77Dw6+txYCTDx7NZ47bl5lTqzGzfJcnIiIistdJN1CPARYlvt8BVCW+fxL4bl8XJZmLxZz75q/m+08tobk9wpUz9+PS4yczrqos36WJiIiI7NXSDdRrgLHAamAZcAqwADgWaOuf0iRdC9dt4/rHFvLGmq0ct98Ibph1KFNHV+a7LBEREZFBId1APZd4iP478DPgt2Z2OfELFG/qp9qkF01tYX74p6Xc/cpKRlQUc9P5R3D2keM1tENEREQkh9IK1O7+taTvHzCzdcDxwFJ3f6y/ipOe1dXv4Mq7a1m5pZmLZ+zLtR87iGFloXyXJSIiIjLo7DZQm9nJ7v7n7u3u/iLwYr9VJbv13Dsb+fJ9r1NcFOD+K49jxn775LskERERkUEr0Mvzz5jZcjO73szG56Qi2a07XlzB5XfVsm91OXOv+ZDCtIiIiEie9RaoDwEeAa4BVprZH8zsbDML9n9p0t0dL65gzhOL+di0MTx01fGM1wweIiIiInm320Dt7m+7+7XABOJzUTvwELDOzL5nZgdlsjEzO9XMlphZnZldt5vlPmlmbmY1max/b3b3KyuZ88RiTjt0DD+76CjKinVMIyIiIlIIejtDDYC7R9z9EXc/A9gX+CnwCWCxmb2QzjoSZ7VvAU4DpgEXmtm0FMtVAl8iPqOIAPfPW823fr+IU6aN5icXHEUomNavTURERERyIONk5u7rgf8jHqq3Aiek+aPHAnXuvtzdO4D7gVkplrsB+D6a3xqAl+s2c/1jC/nwgSO5+aKjKC5SmBYREREpJBmlMzP7JzO7F1gPfId4KE53WMZ44jeI6bQ20Za8/qOAie7+RC91zDazWjOr3bRpU9r151pjcwert7SweksLaxpacPeMfn71lha+cO9rTKmu4OaLjqKkSMM8RERERApNr/NQm9kk4DLgUuLDPV4AZgO/c/dMziKnuttIV8I0swDxm8Rc2tuK3P1W4FaAmpqazFJqDjS3R/jpn9/l9hdXEIm9X97MqdXcfOF0hpX3Pl/0jvYIV9w9H3f41WdrqCzVHNMi/t0FwgAAIABJREFUIiIihai3eaifAU4C6oG7gNvdvW4Pt7UWmJj0eALxM92dKoFDgecTd/obA8w1s7PcvXYPt5lzTy96j+/MXcT6bW2cVzOBGVPi09pt2NbKT/78LrNueZHbPlvD1NGVtEeivLtxBxOHl+8Usre3hfnCb19j2aZm7rrsWCZXV+Tr5YiIiIhIL3o7Q91K/OLDP7h7NMttzQemmtkUYB1wAXBR55Puvg2o7nxsZs8D1w6UMB2Jxvjuk29z50sr+cCYSn564VHUTB6x0zIf3H8frvrNa5zzfy8zubqcJe81EY46VeUhvn7qBzi/ZiIrtzRzxd21rN7Swv9+4jA+NLW6hy2KiIiISCHYbaB297P6akPuHjGzq4GngSBwh7svMrM5QK27z+2rbeVaY3MHX7z3NV5etoXPnTCFb5z+gZQzcRy97wgev+YErn90IR2RGFfM3I8DRw/hvnlr+MYjb3HfvNWs2NxMKBjgt1fM4DjdtEVERESk4FmmF8oVmpqaGq+tzd9J7O1tYc782Yts2NrGd885lE/VTOz9h7pxdx55bR3/8+TbjBpayq2fOZqJI8r7oVoRERER2VNmtsDdd5mQo9eLEmX3Xnp3M6u2tHD7JTWcfPDoPVqHmXHu0RM444ixFAUCBAOprt8UERERkUKkQJ2l+SsbKSkKMHPqyKzXpWnxRERERAYe3SUkS7WrGjhyYpVuuCIiIiIySCkFZqG5PcKi9ds5pttsHiIiIiIyeChQZ+H1NVuJxpyaycPzXYqIiIiI5IkCdRbmr2zADKbvq0AtIiIiMlgpUGdhwapGPjBmKEN1W3ARERGRQUuBeg9FojFeW9XIMRruISIiIjKoKVDvoXfea6K5I7rL7cVFREREZHBRoN5D81c2AOgMtYiIiMggp0C9h2pXNjK+qoyxw8ryXYqIiIiI5JEC9R5wd+avbNB0eSIiIiKiQL0n1jS0Ut/UrvHTIiIiIqJAvSc2N7dz0OhKjZ8WEREREYryXcBANH3ScJ7+1xPzXYaIiIiIFACdoRYRERERyYICtYiIiIhIFszd811DVsxsE7AqT5uvBjbnadsDkforM+qvzKnPMqP+ypz6LDPqr8ypzzKT6/7a191Hdm8c8IE6n8ys1t1r8l3HQKH+yoz6K3Pqs8yovzKnPsuM+itz6rPMFEp/aciHiIiIiEgWFKhFRERERLKgQJ2dW/NdwACj/sqM+itz6rPMqL8ypz7LjPorc+qzzBREf2kMtYiIiIhIFnSGWkREREQkCwrUIiIiIiJZUKAWEREREcmCArWIiIiISBYUqEVEREREsqBALSIiIiKSBQVqEREREZEsKFCLiIiIiGRBgVpEREREJAsK1CIiIiIiWVCgFhERERHJggK1iIiIiEgWFKhFRERERLJQlO8CslVdXe2TJ0/OdxkiIiIispdbsGDBZncf2b19wAfqyZMnU1tbm+8yRERERGQvZ2arUrVryIeIiIiISBYG/BlqEel/7k59Uzubmtq72kLBAMPLQ1SVF1NcpGNzEREZvBSoRQapWMzZ0tzBhm2trN/ayvqtbazf2sqGbW1sbwt3Lbe1JcyKzc3saI/0uK6yUJDy4iCloSAloQCWaDczSkMBykLx50pDQco6v4o72wIELP4T7tAeidIajtIWjhEKWtfPlhUHE98HiMZILBMlYEZZKNC1TGnS+jt/NhQ0rKuq90Visa71dES8q70oaF0HC0NLQwQSPxpzaItEaeuI0h6JUdK53VCQUHDXg4pYzAkEdt2uiIjsXXIaqM3sVOAnQBD4lbvfmGKZ84BvAw684e4X5bJGkULn7mzY1sbyTc0s27SD5Zt2sHxzMys2NwN0hcmhpSGGVxQzvDxEOBqjobmDxuYwjS0dia8w0ZjvtO7SUIBxVWUMLQ2RyLhUlYc4d/p49h81hNFDS7tiaUc0xtaWMI3NHWxtDdMWjgfh9khsp1rbwjHawlF2tEfY1NTetVxbOB5mO5KWBygKxEN0SShAJOa0duy8zkKVXHd7JP6aw1GnLBRkREUxVeWhlKEboKIkyPDyYkZUFDOkpKjrd1hZWkRVor0sFOxavjQUoKq8mKqyEEU9rLO7aMxpC0eJRJ2SUICSogBmRizmtEdidCQOEDrbRUQkfTkL1GYWBG4BTgHWAvPNbK67L05aZirwDeAEd280s1G5qk8kF5rawqzf2sbG7W00tnTEQ24ilDa2dNDcHmFYWTwIDykpoqktklgm8dUcZvOO9p0CZkVxkP1GDmH6pOEUBYzWcJSWjijb28KsbWyhobmDklCQEeXxUHfAqCFdQXtUZSnjqsoYO6yU8VVlVJWHch6mYjEnOdYHU5zRjcWctkiU1o54GC8KBLrCq/v7Z6tbw/Fldvo+EiPcQyAPBqzrzHbyWexwNMbW1vjvpaktjCcKNCNp+QAdieD8/vZitEWiFAcDlBUHKQ4GaG6P0NDSwdaWMJFuBzAQP+hobo+wYet2tjR30NIRIRzddbmeDCkpStS08ycBRvzThYaWDra1hnc5cDGj6zV0V1L0/qcGuzwXClBaFO+D3k6+O/G+bO2I0R6OgtFVY1Ew1WcGEEkE/9aO6E79EP/UIL7fDi0LUV7c+UlEESMqQgwvL6aqvDjRFqCkKEjyS+j8JKE01HPdlaUhhpWFdtoHYzEn5u/XEQxYv/2NuPtOB7nRpAPSjkiMkqIApcU9fyJSCNydmKf+O+5JNOa4p7/Pd5fL30k6zCyj1y97h1yeoT4WqHP35QBmdj8wC1ictMyVwC3u3gjg7vU5rE+kz4SjMd7b1sa79U3UrmykdlUjb2/YTlPbrsMmzGBYWYgR5cWUlwRZtqk5HuTaIwwtLUqE32JGVZZy4OhK9qkoZtI+Few/soL9Rw5hVGXJgD6jmM6QiEDAKC8uorw49VtWWXEwZftAFY7Gz943tUVobI4feLWFo13Pt0VibE0ckG1vjXQNQ2lNOpBwYHJ1OdMrquIBNFREaShAUTBAe+ewlWisKxx3huvWcJT2cJRUEcLdu5ZpDcd2Cpo9KQ4Guob2ALQlDvhSHVwABM26hg+Fgu8HpY7O19wSZltLB/Xb22kNR2luj7C1dddPW/aUGQwtDcU/XUmcue/+fGefdQ5Bih8gBFIeIPSk8xODroPBxMFfuq/j/U9Eej+wSVcoGEj5eiKxWOJANUY4mvrgNPn1xLzz9x6gJHFw113MoT2xfE/7Qros+UCtjzqj++vJVKa/n1DiILwsFEwrjBcFrGuIW09D2pJFYrEePxUciAIGj3zhhHyXsZNcBurxwJqkx2uBGd2WORDAzF4iPizk2+7+VPcVmdlsYDbApEmT+qVYkXS1dkRZsKqR2lUNLFjVyNKNTdQ3tXed1SwKGIeMH8bZR45nwvAyxlWVMWZYKSMSQbn7GbFO7j6gg7LsuVAwQCgYYGhpiPFVZfkup+C5O9vbImxrCXcF1Lakg4LOsfmdASnVcYB7/BOkhpYwW1s6CJh1HQgEO8f4A5FobKchS50HMJmGwqCRdB3B+9cWFBclX4PQ+ycireFoorLshaOe8vV0hsP3D3J2/dn4tQzxWoMBoz0SSwzXiu66cPzVdV1fUVK05wcFyb+Tlo5oWgd56eh8PaXdfifp1tQeSf/34x4fQtd5YJXOB1SRaIzm9gibd3T0eJCTLGhGaXGQ0qIAlaVFA/7/lkL8ACCXgTrVy+++2xQBU4GPABOAv5nZoe6+dacfcr8VuBWgpqamb/56RDLQHonyt6WbmfvGep59eyMtHVHM4KDRlcycOpLxVWWMqypl8j4VHD6hao/OoA70NzyRXDEzhpXFh2uIiORDLgP1WmBi0uMJwPoUy7zq7mFghZktIR6w5+emRJHUojHniTfX8+K7m1m0fjvv1jcRjjrDy0OcfdR4Tpk2mqP3Hc7QUv2HLiIiMtjkMlDPB6aa2RRgHXAB0H0Gj8eAC4Ffm1k18SEgy3NYo8hO3J3nl2zie0+9wzvvNbFPRTHTxg3lxAP3Y8Z+I/jQAdUFe3GQiIiI5EbOArW7R8zsauBp4uOj73D3RWY2B6h197mJ5/7ZzBYDUeBr7r4lVzWKdNrU1M4fF27gkdfW8fqarey7Tzk3X3QUpx86VvMKi4iIyE4sm6lqCkFNTY3X1tbmuwzZSyxY1cDNz9Xx16WbiHl8TPTFx03i/GMm6W6AIiIig5yZLXD3mu7tulOiCPCP1Y386Jml/O3dzYyoKObzH9mfs44Yz0FjKvNdmoiIiBQ4BWoZ9B5/Yz1feeB1hpWF+MZpH+AzH9y3x/mORURERLpTapBBrTNMHz1pOLdfWkOlZukQERGRDClQy6CVHKbvvOwYKkr05yAiIiKZU4KQQScWc27+Sx03PbuUY/YdoTAtIiIiWVGKkEGlsbmDf33wdZ5fsolZR47jfz9xmMZLi4iISFbSShJmNhtocvf7urVfCAxx99v6oziRvtIWjvK7BWu55S91bNnRwQ1nH8rFMybp9t4iIiKStXRPzV0LXJWifR1wW+JLpOCEozFufWE5d7y4gi3NHRwxYRi/uPhojphYle/SREREZC+RbqCeBKxI0b468ZxIwWntiPLFe1/juXfqOemgkVz14f2ZMWWEzkqLiIhIn0o3UNcDhwEru7UfAejW4FJwtrWGueKu+dSuauR/zjmMi2bouE9ERET6R7qB+n7gp2a2FXgx0TYT+DHwQH8UJrKn1jS0MPs3C6irb+LmC6fz8cPH5rskERER2YulG6i/CUwF/gp0JNpCwOPAf/RDXSIZc3cerF3DnMcXEzDjjkuPYebUkfkuS0RERPZyaQVqd28HzjGzQ4GjAANec/eF/VmcSLq2tYT5t4fe4Nm3N/LB/fbhB+cdwfiqsnyXJSIiIoNAutPmBQBLBOiFSe1BwN091k/1ifRqy452PnP7POrqd/DNM6Zx2fGTCQR04aGIiIjkRiDN5X4HfDVF+1eAh/quHJHM1G9v4/xbX2XZph3cdkkNl39oisK0iIiI5FS6gXom8FSK9qeBD/VdOSLp27i9jfN++Qrrt7Zy1+eO5cMHary0iIiI5F66FyUO4f2LEZNFgKF9V45IesLRGF+45zU2NbXzm8tncPS+w/NdkoiIiAxS6Z6hXgicl6L9PGBxuhszs1PNbImZ1ZnZdSmev9TMNpnZ64mvK9Jdtwwu3/vjOyxY1ciN5x6uMC0iIiJ5le4Z6v8Gfmdmk4HnEm0nAxcD56ezgsQFjLcApwBrgflmNtfduwfyB9z96jTrkkHoqYUb+NWLK7j0+MmcecS4fJcjIiIig1xaZ6jd/ffAp4jfGfHXia/DgfPd/dE0t3UsUOfuy929g/jNYmZlWrAMbnX1TXztoTc5cmIV/3H6wfkuR0RERCTtM9S4+2PAY1lsazywJunxWmBGiuXONbMTgaXAv7r7mu4LmNlsYDbApEm6pfRg8dbabVx65zxKQgFu+fR0iovSHbEkIiIi0n/2OJGY2Rgzu87Mlqb7IynavNvjx4HJ7n448CxwV6oVufut7l7j7jUjR2pmh8HglWVbuPC2VykNBXnwqg/qpi0iIiJSMDIK1GYWMLMzzez3wGrgSiDdIR9rgYlJjycA65MXcPctibsyAtwGHJ1JfbJ3enrRe1xy5zzGDivl4c8fz34jh+S7JBEREZEu6d4pcX/gcuAS4iF8JPDJxDCQdM0HpprZFGAdcAFwUbftjHX3DYmHZwFvZ7B+2Qs9WLuG6x5+k8MnVHHnpccwvKI43yWJiIiI7GS3Z6jN7NNm9jzwJjAFuIL4WWYnPsY5be4eAa4mfjOYt4EH3X2Rmc0xs7MSi33JzBaZ2RvAl4BLM9mG7F1ufWEZ//67NznhgGruuWKGwrSIiIgUJHPvPow56UmzCHAjcKO770hqDwNHpJjyLudqamq8trY232VIH6qrb+KmZ9/lD29u4OOHj+Wm847UBYgiIiKSd2a2wN1rurf3NuTjDuJnlT9sZr8hPkf0tv4oUGT1lhZ+8KclPP7mespDQb588lS+dPJUgoFU17OKiIiIFIbdBmp3n21mXyF+85bLgZ+Y2ZPEZ+zQKUPpM8+9s5Ev3/86sZjz+Q/vzxUz92OEhniIiIjIANDrRYnu3gLcCdxpZgcDnwM2Ay+Z2ePAQ4kbv4hkLBZzfvZcHT/+81KmjR3KLy4+mokjyvNdloiIiEjaMjrL7O5vu/vXiE959zlgH+B3/VGY7P22t4WZ/ZsF3PTsUs45cjwPf/54hWkREREZcNK+U2KyxIwdDwMPm9n4vi1JBoO6+iZm372A1Q0tfPvMaVxy/GTMNFZaREREBp49CtTJ3H1dXxQig8dTCzfwbw++QVlxkHuumMGM/fbJd0kiIiIieyzrQC2SrmjM+dEzS7jlL8s4YmIVv7h4OmOH6RbiIiIiMrApUEtObG3p4Mv3v85fl27igmMm8p1Zh1BSFMx3WSIiIiJZU6CWfvfetjYu+tWrrGlo4X/OOYyLZkzKd0kiIiIifSatWT7M7EkzG5aivTIxL7VISmsaWvjUL1+mfns7v718hsK0iIiI7HXSPUP9MaAkRXspcErflSN7kxWbm7notldp6YhyzxUzOGJiVb5LEhEREelzuw3UZjat81vgQDOrTno6CJwKrO+n2mQAW9PQwgW3vkI46tx35XFMGzc03yWJiIiI9IvezlAvBDzx9dduzxnQAXylH+qSAay+qY2Lb/87beEYD1x1HB8YozAtIiIie6/eAvXBxIPzYmAm8VuOd+oANrh7Wz/VJgPQtpYwn719HvXb27nnyhkK0yIiIrLX222gdvclAGZW5u7tuSlJBqrG5g4u+/V8lm3awR2XHsP0ScPzXZKIiIhIv0trlg/gNDP7aOcDM/t3M6szs9+b2ch+qk0GkDUNLZz785dZvGE7N180nZlTtVuIiIjI4JBuoP5voBjAzI4AbgDuBkYAP+yf0qSQhaMxtuxoZ21jCy/Vbeac/3uZLc0d/PbyGXzskDH5Lk9EREQkZ9KdNm8y8E7i+08Av3f3OWb2BJD2PNRmdirwE+IzhPzK3W/sYblPAg8Bx7h7bbrrl9xYuG4bl945j807OrraxleVcf/sGRwwqjKPlYmIiIjkXrqBugMoT3x/MvGz0wANQFpXnZlZELiF+LzVa4H5ZjbX3Rd3W64S+BLw9zRrkxxauG4bF9/+dyqKi/jOWYdQFgpSVhzkQwdUM7yiON/liYiIiORcuoH6JeB7ZvYCcCxwQaJ9KrAuzXUcC9S5+3IAM7sfmEV8BpFkNwDfB65Nc72SI8lh+r4rj2PSPuW9/5CIiIjIXi7dMdTXEL8r4hXAl919baL9LODPaa5jPLAm6fHaRFsXMzsKmOjuT+xuRWY228xqzax206ZNaW5e9kQ4GuP5JfV89cHXOe+XryhMi4iIiHST1hlqd19JiluMu/s1GWzLUq2660mzAHATcGka9dwK3ApQU1PjvSwue+jNtVu54q5a6pvaqSwt4ozDx/Klk6cyYbjCtIiIiEindId8YGYh4GPA/sCd7r7dzCYC29x9exqrWAtMTHo8gZ1vW14JHAo8b2YAY4C5ZnaWLkzMvdqVDVx253yGlYe47bM1nHhgNSVFwXyXJSIiIlJw0grUZjYZeAYYTfzixMeB7cC/AWXAVWmsZj4w1cymEB93fQFwUeeT7r4NqE7a5vPAtQrTufdy3WYuv6uWscNKuefKGYwdVpbvkkREREQKVrpnqH9C/MLEK4nP7NHpUeD2dFbg7hEzuxp4mvi0eXe4+yIzmwPUuvvc9MuWvla/vY1n367n2bc38rd3NzGluoLfXjGDUZWl+S5NREREpKClG6hPAE5w93BiOEanVcC4dDfm7k/Sbd5qd/9WD8t+JN31yp5pj0T506KNPFi7hhfrNuMOE4aX8ZnjJnP1Rw9ghKbBExEREelVuoE6mPjqbgLQ1HflSH+Lxpx5Kxp4/M31PPnWBra2hBk3rJRrTjqA0w8fy0GjK+l20CQiIiIiu5FuoH6G+NR5n088djOrAP4LeKo/CpO+4e4s2dhE7cpGFqxq5KW6zdQ3tVNeHOSUaaP55NETOH7/aoIBhWgRERGRPZFuoL6W+OwbbxKfj/pu4EDiZ6c/00+1SRbcneeXbOLHzy7ljbXbAKgeUsKxU4Zz2qFjOfngUZQXpz3Ji4iIiIj0IN15qFeb2eHEw/PRxG8I8wBwl7tryEeBWbCqkTmPL+KNtduYMLyMG2YdwokHjmTSiHIN5xARERHpY7sN1GZ2B/E7Iza5+w7g57kpS/ZERyTGj59dyi/+uowxQ0v53rmH8YnpEwgF070hpoiIiIhkqrcz1JcA16ELDwveglWNfPOxhSzesJ3zaybyzTOnMaREQzpERERE+ltviUvjAzLg7jzx5gb+sXoryzfvYG1jKx87ZDRXnzSVsuK+v8tgLOb8ZUk9v/zrcuatbKB6SDG3fbaGU6aN7vNtiYiIiEhq6ZzC9H6vYi9x77zVXP/oQspCQaZUVzCqsoRb/rKMuW+sZ86sQznpoFFZb6MjEuPlZZt59u2NPLu4nve2tzG+qoxvnTGN84+ZSIXOSouIiIjkVDrp673eLmRz974//TrALNu0gxueWMzMqdXcddmxBBLT0L2ybAv/+dhbXHbnfMZXlXHIuKEcMm4YMw+s5qiJVRldJLjkvSauue81lm7cQXlxkBOnjuT0w8dy2qFjNE5aREREJE/MvecT0GYWI3678a27W4m7P9zHdaWtpqbGa2tr87V5AMLRGOf+/GVWN7Tw9FdOZPTQnW/X3RGJ8cD81fx9RQOL129nxZbmrrsSnnnEOM6vmcjk6ooe1+/u3DtvNXMeX0xlaYjvnHUIJx88itLQoD+OEREREckZM1vg7jW7tKcRqMe4e31/FpeNQgjUP/zTEn72XB2/uHg6px46ttflt7WGeXbxRua+sT5xy2/n7KPGc81HpzIlKVg3Nnfwx4Xv8eg/1jJ/ZSMzp1bzo/OOZGRlSX++HBERERFJoadA3duQD42f7sWqLc3c8pc6PnX0hLTCNMCwshDnHj2Bc4+eQH1TG7e9sJzfvLqKx/6xjv1HDsEM3GHF5mYiMWe/6gq+feY0PvvByV1DSURERESkMGiWjyy9VLeFmMMXTjpgj35+VGUp1398GrNP3J87XlrBys3NXc999OBRnHn4OA4ZN1Q3ZBEREREpULsN1O6uK916MX9lA9VDSpi8T3lW6xlZWcLXT/1AH1UlIiIiIrmiwJyleSsaOHbKcJ1BFhERERmkFKizsH5rK+u2tnLM5BH5LkVERERE8kSBOgvzVzYAKFCLiIiIDGI5DdRmdqqZLTGzOjO7LsXz/2Jmb5nZ62b2oplNy2V9mfr7igYqS4o4eOzQfJciIiIiInmSs0BtZkHgFuA0YBpwYYrAfK+7H+buRwLfB36Uq/r2xPwVDUzfdzhBTWUnIiIiMmjl8gz1sUCduy939w7gfmBW8gLuvj3pYQUFPA92Y3MH79bv4NgpGu4hIiIiMpj1Ng91XxoPrEl6vBaY0X0hM/si8FWgGPhoqhWZ2WxgNsCkSZP6vNB0aPy0iIiIiEBuz1CnGhexyxlod7/F3fcHvg78Z6oVufut7l7j7jUjR47s4zLTM39lA8XBAIdPGJaX7YuIiIhIYchloF4LTEx6PAFYv5vl7wfO7teKsjBvZSNHTBxGaSiY71JEREREJI9yGajnA1PNbIqZFQMXAHOTFzCzqUkPPw68m8P60tbSEWHRum0aPy0iIiIiuRtD7e4RM7saeBoIAne4+yIzmwPUuvtc4Goz+ycgDDQCl+Sqvkz8Y/VWIjHX+GkRERERwdwLdiKNtNTU1HhtbW1Ot9nSEWHBqkamTxpORUkur+sUERERkXwxswXuXtO9XWlwD5QXFzFzan4uhhQRERGRwqJbj4uIiIiIZEGBWkREREQkCwN+DLWZbQJW5Wnz1cDmPG17IFJ/ZUb9lTn1WWbUX5lTn2VG/ZU59Vlmct1f+7r7LuN+B3ygziczq001MF1SU39lRv2VOfVZZtRfmVOfZUb9lTn1WWYKpb805ENEREREJAsK1CIiIiIiWVCgzs6t+S5ggFF/ZUb9lTn1WWbUX5lTn2VG/ZU59VlmCqK/NIZaRERERCQLOkMtIiIiIpIFBWoRERERkSwoUO8BMzvVzJaYWZ2ZXZfvegqNmU00s7+Y2dtmtsjMvpxo/7aZrTOz1xNfp+e71kJiZivN7K1E39Qm2kaY2TNm9m7i3+H5rrMQmNlBSfvR62a23cy+on1sZ2Z2h5nVm9nCpLaU+5TF/TTxvvammU3PX+X50UN//T8zeyfRJ4+aWVWifbKZtSbta7/IX+X500Of9fh3aGbfSOxjS8zsY/mpOn966K8HkvpqpZm9nmjXPsZuM0VBvZdpDHWGzCwILAVOAdYC84EL3X1xXgsrIGY2Fhjr7q+ZWSWwADgbOA/Y4e4/yGuBBcrMVgI17r45qe37QIO735g4eBvu7l/PV42FKPE3uQ6YAVyG9rEuZnYisAO4290PTbSl3KcSoeca4HTiffkTd5+Rr9rzoYf++mfgOXePmNn3ABL9NRl4onO5waqHPvs2Kf4OzWwacB9wLDAOeBY40N2jOS06j1L1V7fnfwhsc/c52sfidpMpLqWA3st0hjpzxwJ17r7c3TuA+4FZea6poLj7Bnd/LfF9E/A2MD6/VQ1Ys4C7Et/fRfxNRHZ2MrDM3fN1x9SC5e4vAA3dmnvap2YR/0/e3f1VoCrxH9mgkaq/3P1P7h5JPHwVmJDzwgpYD/tYT2YB97t7u7uvAOqI/586aOyuv8zMiJ94ui+nRRW43WSKgnovU6DO3HhgTdLjtSgs9ihxhH0U8PdE09WJj2Du0PCFXTjwJzNbYGazE22j3X0DxN9UgFF5q65wXcDO/wFpH9u9nvYpvbf17nPAH5MeTzGzf5j5Ff8nAAACv0lEQVTZX81sZr6KKlCp/g61j+3eTGCju7+b1KZ9LEm3TFFQ72UK1JmzFG0aN5OCmQ0BHga+4u7bgZ8D+wNHAhuAH+axvEJ0grtPB04Dvpj4aFB2w8yKgbOAhxJN2sf2nN7bdsPMrgciwD2Jpg3AJHc/CvgqcK+ZDc1XfQWmp79D7WO7dyE7nxzQPpYkRabocdEUbf2+nylQZ24tMDHp8QRgfZ5qKVhmFiK+49/j7o8AuPtGd4+6ewy4jUH2UV9v3H194t964FHi/bOx86OqxL/1+auwIJ0GvObuG0H7WJp62qf03tYDM7sEOAP4tCcuPEoMW9iS+H4BsAw4MH9VFo7d/B1qH+uBmRUBnwAe6GzTPva+VJmCAnsvU6DO3HxgqplNSZwduwCYm+eaCkpiHNjtwNvu/qOk9uQxTOcAC7v/7GBlZhWJiy0wswrgn4n3z1zgksRilwC/z0+FBWunMzrax9LS0z41F/hs4gr544hfGLUhHwUWEjM7Ffg6cJa7tyS1j0xcEIuZ7QdMBZbnp8rCspu/w7nABWZWYmZTiPfZvFzXV6D+CXjH3dd2Nmgfi+spU1Bg72VF/b2BvU3iSu+rgaeBIHCHuy/Kc1mF5gTgM8BbndP/AP8BXGhmRxL/6GUlcFV+yitIo4FH4+8bFAH3uvtTZjYfeNDMLgdWA5/KY40FxczKic+2k7wffV/72PvM7D7gI0C1ma0F/gu4kdT71JPEr4qvA1qIz5gyqPTQX98ASoBnEn+fr7r7vwAnAnPMLAJEgX9x93Qvzttr9NBnH0n1d+jui8zsQWAx8eEzXxxMM3xA6v5y99vZ9VoQ0D7WqadMUVDvZZo2T0REREQkCxryISIiIiKSBQVqEREREZEsKFCLiIiIiGRBgVpEREREJAsK1CIiIiIiWVCgFhERERHJggK1iIiIiEgW/j9rSBjlmXD1EAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x576 with 4 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 训练过程可视化\n",
    "fig, axes = plt.subplots(4, sharex=True, figsize=(12, 8))\n",
    "fig.suptitle('Training Metrics')\n",
    "\n",
    "axes[0].set_ylabel(\"Loss\", fontsize=14)\n",
    "axes[0].plot(train_loss_results)\n",
    "\n",
    "axes[1].set_ylabel(\"Accuracy\", fontsize=14)\n",
    "axes[1].plot(train_accuracy_results)\n",
    "\n",
    "axes[2].set_ylabel(\"Val Acc\", fontsize=14)\n",
    "axes[2].plot(train_val_results)\n",
    "\n",
    "axes[3].set_ylabel(\"Test Acc\", fontsize=14)\n",
    "axes[3].plot(train_test_results)\n",
    "\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
